00000800:00000200:3.1F:1775763739.873721:0:10942:0:(socklnd.c:214:ksocknal_find_peer_locked()) got peer_ni [00000000c9f2fd80] -> 12345-192.168.202.105@tcp (2) 00000800:00000200:2.1F:1775763739.878514:0:10932:0:(socklnd.c:384:ksocknal_incr_conn_count()) Add conn type 1, ksnr_connected 2 ksnr_max_conns 1 00000800:00000200:2.1:1775763739.878545:0:10932:0:(socklnd_cb.c:748:ksocknal_queue_tx_locked()) Sending to 12345-192.168.202.105@tcp ip 192.168.202.105:988 00000800:00000200:2.1:1775763739.878550:0:10932:0:(socklnd_cb.c:767:ksocknal_queue_tx_locked()) Packet 00000000ab8436d9 type 2, nob 96 niov 1 nkiov 0 00000800:00000200:3.1:1775763739.891184:0:10932:0:(socklnd.c:384:ksocknal_incr_conn_count()) Add conn type 2, ksnr_connected 6 ksnr_max_conns 1 00000800:00000200:3.1:1775763739.907401:0:10932:0:(socklnd.c:384:ksocknal_incr_conn_count()) Add conn type 3, ksnr_connected e ksnr_max_conns 1 00000002:00010000:1.0:1775764921.657470:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3de9 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215073e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.657476:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3de9 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215073e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.657481:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/28 (0 0) 00000080:00200000:1.0:1775764921.657484:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.657493:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x1c:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.657525:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x1c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.657531:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x1c:0x0](ffff9d8184281148) for lock 0xeef44110eff3de9 00000080:00010000:1.0:1775764921.657534:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x1c:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.657536:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3de9 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.657538:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3de9 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215073e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.657542:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3de9 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215073e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.657551:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.657554:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=oos.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.657570:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3de9 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215073e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.657579:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x1c:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.657592:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3de9 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215073e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.657596:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.657598:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3de9 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215073e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.657608:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.657611:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893fa300 x1862023065985280/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.657624:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893fa300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065985280:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.657653:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.658679:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893fa300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065985280:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.658700:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x1c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.658704:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.658719:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x1c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.658725:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x1c:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.658754:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.658759:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-dom.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.658761:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-dom.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.658767:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-dom.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.658781:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.658785:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.658790:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.658797:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f8700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065985408:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.658820:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.659639:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f8700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065985408:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.659648:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659659:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659666:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659673:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659681:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659689:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659694:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659698:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.659703:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893f8700 x1862023065985408/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.659708:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.659714:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659719:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/59 (0 0) 00000080:00200000:1.0:1775764921.659721:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.659729:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x3b:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.659763:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x3b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.659769:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x3b:0x0](ffff9d81842800c8) for lock 0xeef44110eff3df0 00000080:00010000:1.0:1775764921.659772:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x3b:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.659774:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3df0 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.659776:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.659780:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.659790:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.659794:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-dom.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.659819:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.659836:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x3b:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.659857:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.659864:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.659867:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3df0 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150776 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.659895:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.659899:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893f8700 x1862023065985664/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.659908:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f8700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065985664:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.659930:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.660940:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f8700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065985664:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.660967:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.660973:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.660994:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.661003:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x3b:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.661044:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.661061:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=recovery-random-scale.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.661065:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=recovery-random-scale.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.661072:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: recovery-random-scale.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.661086:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661091:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661095:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.661102:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065985792:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.661124:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.661928:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065985792:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.661935:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661943:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661947:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661951:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661956:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661960:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661964:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661968:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.661973:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893f9c00 x1862023065985792/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 600/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.661978:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.661984:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.661988:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/42 (0 0) 00000080:00200000:1.0:1775764921.661991:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.661999:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x2a:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.662030:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x2a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.662036:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x2a:0x0](ffff9d81842821c8) for lock 0xeef44110eff3df7 00000080:00010000:1.0:1775764921.662039:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x2a:0x0](ffff9d81842821c8) 00000080:00010000:1.0:1775764921.662042:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3df7 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.662045:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.662052:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.662065:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.662070:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=recovery-random-scale.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.662096:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.662112:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x2a:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.662127:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.662131:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.662132:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3df7 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321507ae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.662142:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.662146:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893f9c00 x1862023065986048/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.662154:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986048:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.662177:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.663102:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986048:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.663125:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x2a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.663129:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.663146:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x2a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.663153:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x2a:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.663184:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.663205:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=parallel-scale-cifs.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.663209:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=parallel-scale-cifs.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.663218:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: parallel-scale-cifs.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.663239:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.663247:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.663254:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.663266:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986176:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.663298:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.664078:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986176:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.664085:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664097:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 4/1,0 mode: --/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664104:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 4/1,0 mode: --/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664112:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664120:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664129:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664136:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664143:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.664152:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893f9f80 x1862023065986176/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.664161:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.664170:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664175:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/31 (0 0) 00000080:00200000:1.0:1775764921.664178:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.664200:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x1f:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.664237:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x1f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.664246:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x1f:0x0](ffff9d8184286c08) for lock 0xeef44110eff3dfe 00000080:00010000:1.0:1775764921.664251:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x1f:0x0](ffff9d8184286c08) 00000080:00010000:1.0:1775764921.664253:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3dfe from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.664256:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.664263:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.664278:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.664282:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=parallel-scale-cifs.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.664312:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.664323:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x1f:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.664338:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 1/0,0 mode: --/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.664342:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.664343:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3dfe lrc: 0/0,0 mode: --/PR res: [0x200004282:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321507e6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.664353:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.664357:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893f9f80 x1862023065986432/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.664366:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986432:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.664391:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.665339:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986432:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.665362:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x1f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.665367:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.665385:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.665391:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x1f:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.665422:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.665428:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=pjdfstest.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.665430:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=pjdfstest.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.665437:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: pjdfstest.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.665452:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.665457:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.665461:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.665469:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986560:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.665494:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.666303:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986560:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.666313:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666326:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x25:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666332:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666339:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666347:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666355:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666361:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666368:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.666377:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893f9880 x1862023065986560/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.666384:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.666393:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666401:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/37 (0 0) 00000080:00200000:1.0:1775764921.666405:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.666418:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x25:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.666468:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x25:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.666477:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x25:0x0](ffff9d8184285348) for lock 0xeef44110eff3e05 00000080:00010000:1.0:1775764921.666483:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x25:0x0](ffff9d8184285348) 00000080:00010000:1.0:1775764921.666485:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e05 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.666488:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.666495:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.666509:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.666514:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=pjdfstest.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.666541:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.666557:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x25:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.666577:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.666583:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.666585:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e05 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215081e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.666599:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.666604:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893f9880 x1862023065986816/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.666616:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986816:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.666646:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.667718:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986816:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.667751:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.667756:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.667782:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.667793:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x25:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.667837:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.667846:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-pcc.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.667850:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-pcc.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.667859:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-pcc.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.667900:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.667908:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.667916:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.667930:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f8a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986944:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.667960:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.668808:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f8a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065986944:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.668819:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668831:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 4/1,0 mode: --/PR res: [0x200004282:0x43:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668836:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 4/1,0 mode: --/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668842:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668847:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668853:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668857:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668861:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.668866:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893f8a80 x1862023065986944/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.668872:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.668914:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668920:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/67 (0 0) 00000080:00200000:1.0:1775764921.668922:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.668932:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x43:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.668972:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x43:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.668977:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x43:0x0](ffff9d8184281148) for lock 0xeef44110eff3e0c 00000080:00010000:1.0:1775764921.668981:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x43:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.668983:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e0c from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.668985:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.668992:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.669007:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.669011:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-pcc.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.669039:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.669056:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x43:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.669079:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 1/0,0 mode: --/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.669085:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.669088:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e0c lrc: 0/0,0 mode: --/PR res: [0x200004282:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150856 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.669103:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.669109:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893f8a80 x1862023065987200/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.669123:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f8a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987200:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.669156:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.670297:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f8a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987200:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.670323:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x43:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.670328:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.670349:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x43:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.670357:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x43:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.670391:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.670397:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=mmp_mark.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.670399:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=mmp_mark.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.670406:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: mmp_mark.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.670421:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.670426:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.670431:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.670438:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987328:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.670465:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.671302:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987328:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.671312:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671323:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x18:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671330:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671339:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671347:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671355:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671362:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671368:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.671376:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893f8380 x1862023065987328/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.671385:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.671393:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671400:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/24 (0 0) 00000080:00200000:1.0:1775764921.671404:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.671416:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x18:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.671466:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x18:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.671475:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x18:0x0](ffff9d8184282a08) for lock 0xeef44110eff3e13 00000080:00010000:1.0:1775764921.671480:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x18:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.671483:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e13 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.671486:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.671493:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.671516:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.671521:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=mmp_mark.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.671548:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.671564:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x18:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.671584:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.671590:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.671593:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e13 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215088e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.671607:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.671613:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893f8380 x1862023065987584/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.671625:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987584:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.671654:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.672750:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987584:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.672777:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.672782:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.672802:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.672809:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x18:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.672844:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.672850:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=rpc.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.672852:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=rpc.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.672859:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: rpc.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.672889:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.672896:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.672902:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.672911:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987712:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.672937:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.673800:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987712:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.673808:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673818:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 4/1,0 mode: --/PR res: [0x200004282:0x32:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673823:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 4/1,0 mode: --/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673828:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673833:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673838:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673843:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673847:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.673852:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893f9180 x1862023065987712/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.673857:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.673863:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673868:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/50 (0 0) 00000080:00200000:1.0:1775764921.673870:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.673917:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x32:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.673960:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x32:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.673966:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x32:0x0](ffff9d81842821c8) for lock 0xeef44110eff3e1a 00000080:00010000:1.0:1775764921.673970:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x32:0x0](ffff9d81842821c8) 00000080:00010000:1.0:1775764921.673972:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e1a from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.673974:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.673980:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.673993:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.673998:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=rpc.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.674025:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.674044:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x32:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.674065:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 1/0,0 mode: --/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.674069:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.674071:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e1a lrc: 0/0,0 mode: --/PR res: [0x200004282:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321508c6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.674082:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.674086:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893f9180 x1862023065987968/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.674096:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987968:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.674120:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.675176:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065987968:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.675212:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.675217:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.675241:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.675251:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x32:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.675297:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.675306:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=ha.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.675310:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=ha.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.675320:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ha.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.675340:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.675347:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.675364:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.675376:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988096:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.675407:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.676366:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988096:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.676373:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676382:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676386:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676392:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676396:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676401:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676405:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676409:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.676414:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893fb480 x1862023065988096/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.676419:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.676425:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676430:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/9 (0 0) 00000080:00200000:1.0:1775764921.676432:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.676442:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x9:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.676477:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x9:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.676482:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x9:0x0](ffff9d81842800c8) for lock 0xeef44110eff3e21 00000080:00010000:1.0:1775764921.676486:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x9:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.676488:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e21 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.676490:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.676498:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.676517:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.676521:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=ha.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.676550:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.676564:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x9:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.676580:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.676584:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.676585:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e21 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321508fe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.676596:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.676599:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893fb480 x1862023065988352/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.676608:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988352:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.676635:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.677604:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988352:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.677630:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.677635:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.677661:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.677672:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x9:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.677718:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.677725:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=replay-ost-single.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.677734:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=replay-ost-single.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.677741:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: replay-ost-single.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.677758:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.677765:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.677772:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.677783:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893fad80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988480:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.677816:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.678691:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893fad80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988480:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.678700:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678712:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678719:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678728:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678736:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678744:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678751:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678758:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.678768:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893fad80 x1862023065988480/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.678777:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.678784:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678790:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/46 (0 0) 00000080:00200000:1.0:1775764921.678792:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.678805:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x2e:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.678847:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x2e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.678856:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x2e:0x0](ffff9d8184281988) for lock 0xeef44110eff3e28 00000080:00010000:1.0:1775764921.678862:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x2e:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.678865:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e28 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.678868:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.678921:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.678938:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.678943:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=replay-ost-single.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.678965:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.678984:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x2e:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.679002:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.679006:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.679007:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e28 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150936 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.679019:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.679023:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893fad80 x1862023065988736/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.679031:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893fad80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988736:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.679056:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.680178:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893fad80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988736:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.680219:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.680226:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.680251:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.680265:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x2e:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.680302:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.680309:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=recovery-small.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.680312:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=recovery-small.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.680319:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: recovery-small.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.680334:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.680339:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.680344:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.680351:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988864:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.680375:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.681171:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065988864:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.681179:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681199:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681206:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681214:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681223:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681231:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681239:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681246:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.681254:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893f9500 x1862023065988864/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.681259:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.681266:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681270:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/44 (0 0) 00000080:00200000:1.0:1775764921.681273:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.681283:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x2c:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.681327:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x2c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.681337:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x2c:0x0](ffff9d8184285348) for lock 0xeef44110eff3e2f 00000080:00010000:1.0:1775764921.681342:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x2c:0x0](ffff9d8184285348) 00000080:00010000:1.0:1775764921.681346:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e2f from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.681349:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.681357:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.681373:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.681377:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=recovery-small.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.681406:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.681421:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x2c:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.681438:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 1/0,0 mode: --/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.681442:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.681444:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e2f lrc: 0/0,0 mode: --/PR res: [0x200004282:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215096e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.681456:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.681461:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893f9500 x1862023065989120/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.681473:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893f9500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989120:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.681504:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.682522:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893f9500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989120:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.682547:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x2c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.682552:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.682571:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x2c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.682577:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x2c:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.682611:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.682618:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=replay-dual.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.682621:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=replay-dual.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.682628:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: replay-dual.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.682643:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.682648:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.682653:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.682661:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893fa680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989248:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.682685:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.683608:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893fa680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989248:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.683617:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683627:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683634:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683644:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683652:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683661:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683669:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683675:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.683685:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81893fa680 x1862023065989248/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.683695:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.683704:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683713:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/45 (0 0) 00000080:00200000:1.0:1775764921.683716:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.683727:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x2d:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.683774:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x2d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.683785:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x2d:0x0](ffff9d8184282a08) for lock 0xeef44110eff3e36 00000080:00010000:1.0:1775764921.683790:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x2d:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.683792:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e36 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.683794:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.683799:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.683812:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.683815:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=replay-dual.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.683835:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.683852:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x2d:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.683900:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.683908:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.683910:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e36 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321509a6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.683928:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.683933:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81893fa680 x1862023065989504/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.683947:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81893fa680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989504:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.683982:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.684953:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81893fa680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989504:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.684977:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x2d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.684982:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.685002:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x2d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.685009:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x2d:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.685045:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.685051:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup_kerberos.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.685054:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup_kerberos.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.685061:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup_kerberos.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.685078:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.685083:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.685087:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.685095:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c2300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989632:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.685123:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.685935:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c2300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989632:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.685951:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.685964:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 4/1,0 mode: --/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.685971:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 4/1,0 mode: --/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.685980:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.685988:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.685996:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.686004:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.686011:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.686020:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c2300 x1862023065989632/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.686030:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.686040:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.686048:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/76 (0 0) 00000080:00200000:1.0:1775764921.686052:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.686068:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x4c:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.686122:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x4c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.686132:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x4c:0x0](ffff9d8184281148) for lock 0xeef44110eff3e3d 00000080:00010000:1.0:1775764921.686138:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x4c:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.686141:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e3d from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.686144:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.686151:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.686166:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.686171:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=setup_kerberos.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.686217:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.686234:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x4c:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.686259:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 1/0,0 mode: --/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.686265:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.686267:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e3d lrc: 0/0,0 mode: --/PR res: [0x200004282:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321509de expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.686284:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.686290:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c2300 x1862023065989888/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.686303:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c2300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989888:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.686333:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.687309:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c2300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065989888:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.687342:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x4c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.687348:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.687377:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x4c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.687388:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x4c:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.687439:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.687450:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup-nfs.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.687453:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-nfs.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.687463:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup-nfs.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.687483:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.687491:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.687498:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.687521:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990016:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.687558:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.688376:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990016:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.688387:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688400:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688407:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688416:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688425:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688433:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688440:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688446:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.688455:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c0700 x1862023065990016/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.688464:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.688472:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688480:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/77 (0 0) 00000080:00200000:1.0:1775764921.688484:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.688497:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x4d:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.688563:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x4d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.688573:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x4d:0x0](ffff9d81842800c8) for lock 0xeef44110eff3e44 00000080:00010000:1.0:1775764921.688579:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x4d:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.688582:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e44 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.688585:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.688593:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.688608:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.688612:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=setup-nfs.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.688640:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.688656:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x4d:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.688678:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.688685:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.688687:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e44 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150a16 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.688703:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.688709:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c0700 x1862023065990272/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.688722:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990272:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.688755:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.689833:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990272:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.689864:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.689870:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.689916:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.689927:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x4d:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.689982:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.689995:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=mmp.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.689999:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=mmp.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.690008:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: mmp.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.690029:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690036:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690043:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.690056:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c3800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990400:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.690088:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.690951:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c3800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990400:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.690959:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690968:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 4/1,0 mode: --/PR res: [0x200004282:0x19:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690973:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 4/1,0 mode: --/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690978:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690983:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690988:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690992:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.690996:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.691001:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c3800 x1862023065990400/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.691006:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.691012:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.691017:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/25 (0 0) 00000080:00200000:1.0:1775764921.691020:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.691031:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x19:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.691069:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x19:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.691074:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x19:0x0](ffff9d81842821c8) for lock 0xeef44110eff3e4b 00000080:00010000:1.0:1775764921.691079:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x19:0x0](ffff9d81842821c8) 00000080:00010000:1.0:1775764921.691081:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e4b from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.691083:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.691088:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.691098:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.691101:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=mmp.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.691117:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.691127:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x19:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.691142:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 1/0,0 mode: --/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.691146:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.691148:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e4b lrc: 0/0,0 mode: --/PR res: [0x200004282:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150a4e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.691159:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.691163:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c3800 x1862023065990656/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.691172:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c3800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990656:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.691210:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.692256:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c3800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990656:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.692278:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.692282:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.692307:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.692318:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x19:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.692367:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.692376:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=mds-survey.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.692380:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=mds-survey.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.692389:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: mds-survey.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.692410:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.692416:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.692424:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.692436:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c3b80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990784:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.692468:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.693315:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c3b80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065990784:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.693325:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693337:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x17:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693344:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693352:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693360:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693368:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693375:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693381:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.693390:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c3b80 x1862023065990784/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.693399:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.693408:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693416:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/23 (0 0) 00000080:00200000:1.0:1775764921.693420:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.693431:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x17:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.693469:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x17:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.693475:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x17:0x0](ffff9d8184285348) for lock 0xeef44110eff3e52 00000080:00010000:1.0:1775764921.693478:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x17:0x0](ffff9d8184285348) 00000080:00010000:1.0:1775764921.693480:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e52 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.693482:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.693487:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.693497:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.693500:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=mds-survey.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.693522:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.693532:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x17:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.693546:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.693550:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.693552:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e52 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150a86 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.693562:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.693566:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c3b80 x1862023065991040/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.693574:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c3b80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991040:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.693598:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.694577:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c3b80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991040:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.694600:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.694605:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.694624:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.694631:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x17:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.694661:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.694667:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_tar.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.694669:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_tar.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.694675:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: run_tar.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.694705:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.694710:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.694715:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.694723:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c2680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991168:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.694753:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.695618:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c2680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991168:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.695628:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695641:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x38:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695648:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695656:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695665:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695673:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695681:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695688:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.695696:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c2680 x1862023065991168/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.695705:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.695713:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695718:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/56 (0 0) 00000080:00200000:1.0:1775764921.695721:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.695731:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x38:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.695766:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x38:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.695775:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x38:0x0](ffff9d8184281988) for lock 0xeef44110eff3e59 00000080:00010000:1.0:1775764921.695780:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x38:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.695782:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e59 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.695784:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.695792:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.695805:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.695810:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run_tar.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.695835:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.695851:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x38:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.695873:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.695892:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.695894:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e59 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150abe expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.695908:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.695912:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c2680 x1862023065991424/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.695922:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c2680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991424:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.695947:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.696971:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c2680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991424:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.696994:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.696998:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.697015:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.697022:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x38:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.697054:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.697059:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=lfsck-performance.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.697062:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=lfsck-performance.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.697068:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: lfsck-performance.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.697083:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.697088:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.697093:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.697100:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c2d80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991552:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.697125:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.697995:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c2d80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991552:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.698004:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698016:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 4/1,0 mode: --/PR res: [0x200004282:0xf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698023:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 4/1,0 mode: --/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698030:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698037:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698045:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698052:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698058:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.698065:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c2d80 x1862023065991552/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.698073:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.698082:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698089:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/15 (0 0) 00000080:00200000:1.0:1775764921.698093:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.698104:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0xf:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.698157:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0xf:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.698167:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0xf:0x0](ffff9d8184281148) for lock 0xeef44110eff3e60 00000080:00010000:1.0:1775764921.698172:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0xf:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.698174:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e60 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.698177:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.698198:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.698213:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.698218:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=lfsck-performance.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.698244:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.698258:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0xf:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.698277:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 1/0,0 mode: --/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.698283:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.698285:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e60 lrc: 0/0,0 mode: --/PR res: [0x200004282:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150af6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.698299:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.698303:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c2d80 x1862023065991808/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.698315:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c2d80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991808:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.698346:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.699410:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c2d80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991808:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.699434:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.699439:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.699456:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.699463:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0xf:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.699493:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.699501:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-flr.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.699503:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-flr.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.699509:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-flr.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.699524:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.699529:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.699534:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.699541:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991936:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.699567:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.700436:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065991936:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.700446:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700458:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700465:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700473:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700481:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700489:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700496:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700503:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.700511:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c1880 x1862023065991936/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.700520:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.700529:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700537:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/60 (0 0) 00000080:00200000:1.0:1775764921.700541:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.700555:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x3c:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.700607:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x3c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.700615:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x3c:0x0](ffff9d8184282a08) for lock 0xeef44110eff3e67 00000080:00010000:1.0:1775764921.700620:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x3c:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.700622:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e67 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.700625:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.700632:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.700644:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.700648:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-flr.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.700674:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.700690:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x3c:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.700711:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.700717:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.700720:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e67 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150b2e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.700744:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.700750:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c1880 x1862023065992192/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.700763:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992192:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.700794:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.701918:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992192:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.701951:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.701958:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.701986:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.701997:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x3c:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.702043:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.702053:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_dbench.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.702056:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_dbench.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.702066:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: run_dbench.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.702087:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.702094:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.702101:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.702113:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c3100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992320:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.702148:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.703049:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c3100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992320:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.703059:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703071:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 4/1,0 mode: --/PR res: [0x200004282:0x33:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703077:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 4/1,0 mode: --/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703085:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703093:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703100:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703107:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703114:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.703122:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c3100 x1862023065992320/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.703130:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.703139:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703146:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/51 (0 0) 00000080:00200000:1.0:1775764921.703149:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.703163:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x33:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.703234:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x33:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.703243:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x33:0x0](ffff9d81842821c8) for lock 0xeef44110eff3e6e 00000080:00010000:1.0:1775764921.703249:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x33:0x0](ffff9d81842821c8) 00000080:00010000:1.0:1775764921.703252:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e6e from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.703255:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.703263:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.703277:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.703282:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run_dbench.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.703316:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.703334:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x33:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.703356:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 1/0,0 mode: --/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.703362:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.703365:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e6e lrc: 0/0,0 mode: --/PR res: [0x200004282:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150b66 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.703380:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.703386:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c3100 x1862023065992576/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.703399:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c3100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992576:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.703432:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.704557:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c3100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992576:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.704589:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.704596:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.704621:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.704631:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x33:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.704679:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.704688:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_dd.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.704692:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_dd.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.704700:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: run_dd.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.704721:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.704728:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.704735:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.704746:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992704:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.704777:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.705720:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992704:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.705731:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705744:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x34:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705752:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705760:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705768:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705776:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705784:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705790:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.705799:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c1c00 x1862023065992704/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.705808:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.705817:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705824:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/52 (0 0) 00000080:00200000:1.0:1775764921.705828:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.705842:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x34:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.705934:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x34:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.705944:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x34:0x0](ffff9d81842800c8) for lock 0xeef44110eff3e75 00000080:00010000:1.0:1775764921.705950:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x34:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.705953:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e75 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.705956:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.705965:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.705980:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.705985:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run_dd.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.706014:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.706032:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x34:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.706055:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.706062:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.706065:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e75 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150b9e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.706079:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.706085:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c1c00 x1862023065992960/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.706099:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992960:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.706131:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.707279:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065992960:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.707305:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.707311:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.707339:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.707350:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x34:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.707402:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.707411:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-pfl.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.707415:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-pfl.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.707425:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-pfl.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.707447:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.707455:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.707463:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.707472:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993088:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.707498:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.708499:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993088:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.708507:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708516:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 4/1,0 mode: --/PR res: [0x200004282:0x44:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708521:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 4/1,0 mode: --/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708526:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708531:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708536:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708540:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708544:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.708549:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c0000 x1862023065993088/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.708554:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.708560:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708565:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/68 (0 0) 00000080:00200000:1.0:1775764921.708567:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.708578:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x44:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.708614:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x44:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.708620:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x44:0x0](ffff9d8184285348) for lock 0xeef44110eff3e7c 00000080:00010000:1.0:1775764921.708624:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x44:0x0](ffff9d8184285348) 00000080:00010000:1.0:1775764921.708626:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e7c from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.708628:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.708633:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.708642:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.708646:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-pfl.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.708663:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.708672:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x44:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.708691:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 1/0,0 mode: --/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.708695:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.708697:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3e7c lrc: 0/0,0 mode: --/PR res: [0x200004282:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150bd6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.708708:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.708711:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c0000 x1862023065993344/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.708719:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993344:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.708743:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.709868:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993344:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.709917:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.709924:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.709950:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.709961:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x44:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.710006:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.710015:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=functions.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.710018:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=functions.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.710028:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: functions.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.710047:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710055:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710062:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.710074:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0e00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993472:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.710105:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.710861:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0e00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993472:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.710870:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710922:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710929:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710936:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710943:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710951:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710958:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710963:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.710971:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c0e00 x1862023065993472/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.710979:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.710987:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.710993:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/8 (0 0) 00000080:00200000:1.0:1775764921.710997:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.711008:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x8:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.711051:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x8:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.711060:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x8:0x0](ffff9d8184281148) for lock 0xeef44110eff3e83 00000080:00010000:1.0:1775764921.711064:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x8:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.711067:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e83 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.711070:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.711077:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.711089:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.711093:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=functions.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.711116:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.711130:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x8:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.711148:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.711154:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.711157:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3e83 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150c0e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.711169:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.711175:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c0e00 x1862023065993728/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.711199:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0e00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993728:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.711231:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.712383:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0e00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993728:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.712419:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.712424:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.712441:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.712448:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x8:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.712477:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.712483:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-hsm.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.712486:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-hsm.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.712492:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-hsm.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.712515:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.712520:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.712526:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.712537:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c3480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993856:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.712569:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.713473:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c3480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065993856:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.713483:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713496:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 4/1,0 mode: --/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713503:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 4/1,0 mode: --/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713511:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713519:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713528:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713535:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713542:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.713552:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c3480 x1862023065993856/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.713561:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.713568:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713573:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/61 (0 0) 00000080:00200000:1.0:1775764921.713576:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.713586:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x3d:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.713627:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x3d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.713632:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x3d:0x0](ffff9d8184281988) for lock 0xeef44110eff3e8a 00000080:00010000:1.0:1775764921.713636:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x3d:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.713638:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e8a from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.713640:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.713645:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.713655:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.713658:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-hsm.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.713677:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.713694:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x3d:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.713717:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 1/0,0 mode: --/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.713724:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.713735:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3e8a lrc: 0/0,0 mode: --/PR res: [0x200004282:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150c46 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.713752:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.713756:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c3480 x1862023065994112/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.713766:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c3480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994112:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.713791:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.714736:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c3480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994112:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.714758:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.714762:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.714779:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.714786:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x3d:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.714822:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.714830:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=lutf.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.714834:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=lutf.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.714843:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: lutf.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.714863:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.714871:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.714898:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.714909:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994240:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.714934:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.715830:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994240:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.715840:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.715853:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x15:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.715861:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.715869:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.715918:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.715925:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.715929:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.715933:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.715939:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c1500 x1862023065994240/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.715944:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.715950:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.715956:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/21 (0 0) 00000080:00200000:1.0:1775764921.715958:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.715967:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x15:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.716023:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x15:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.716034:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x15:0x0](ffff9d81842821c8) for lock 0xeef44110eff3e91 00000080:00010000:1.0:1775764921.716039:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x15:0x0](ffff9d81842821c8) 00000080:00010000:1.0:1775764921.716049:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e91 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.716052:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.716060:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.716073:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.716076:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=lutf.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.716096:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.716106:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x15:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.716120:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.716124:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.716126:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3e91 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150c7e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.716136:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.716140:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c1500 x1862023065994496/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.716148:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994496:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.716178:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.717310:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994496:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.717350:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.717356:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.717379:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.717386:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x15:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.717421:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.717430:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=insanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.717432:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=insanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.717440:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: insanity.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.717457:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.717463:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.717468:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.717476:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994624:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.717502:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.718496:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994624:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.718504:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718514:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 4/1,0 mode: --/PR res: [0x200004282:0xa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718519:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 4/1,0 mode: --/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718525:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718530:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718535:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718539:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718543:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.718549:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c0380 x1862023065994624/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.718554:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.718561:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718566:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/10 (0 0) 00000080:00200000:1.0:1775764921.718568:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.718579:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0xa:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.718628:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0xa:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.718633:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0xa:0x0](ffff9d8184282a08) for lock 0xeef44110eff3e98 00000080:00010000:1.0:1775764921.718638:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0xa:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.718640:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e98 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.718642:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.718647:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.718658:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.718661:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=insanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.718679:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.718689:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0xa:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.718705:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 1/0,0 mode: --/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.718710:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.718711:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3e98 lrc: 0/0,0 mode: --/PR res: [0x200004282:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150cb6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.718724:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.718728:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c0380 x1862023065994880/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.718737:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994880:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.718765:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.720066:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065994880:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.720102:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.720109:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.720133:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.720141:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0xa:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.720197:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.720205:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=oos2.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.720208:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=oos2.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.720216:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: oos2.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.720233:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.720241:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.720250:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.720263:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995008:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.720295:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.721379:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995008:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.721391:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721405:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 4/1,0 mode: --/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721412:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 4/1,0 mode: --/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721420:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721429:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721445:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721453:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721459:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.721469:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c1f80 x1862023065995008/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.721477:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.721487:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721494:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/27 (0 0) 00000080:00200000:1.0:1775764921.721498:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.721523:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x1b:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.721588:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x1b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.721599:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x1b:0x0](ffff9d8184286c08) for lock 0xeef44110eff3e9f 00000080:00010000:1.0:1775764921.721605:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x1b:0x0](ffff9d8184286c08) 00000080:00010000:1.0:1775764921.721608:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3e9f from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.721611:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.721616:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.721629:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.721633:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=oos2.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.721652:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.721664:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x1b:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.721680:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 1/0,0 mode: --/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.721684:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.721686:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3e9f lrc: 0/0,0 mode: --/PR res: [0x200004282:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150cee expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.721699:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.721704:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c1f80 x1862023065995264/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.721713:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995264:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.721737:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.723141:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995264:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.723171:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.723178:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.723224:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.723236:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x1b:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.723294:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.723302:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=replay-single.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.723304:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=replay-single.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.723313:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: replay-single.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.723333:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.723338:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.723347:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.723358:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995392:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.723394:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.724331:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995392:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.724339:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724352:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x30:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724359:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724367:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724376:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724385:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724393:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724400:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.724406:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c1180 x1862023065995392/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.724412:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.724419:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724424:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/48 (0 0) 00000080:00200000:1.0:1775764921.724427:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.724438:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x30:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.724481:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x30:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.724488:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x30:0x0](ffff9d8184285b88) for lock 0xeef44110eff3ea6 00000080:00010000:1.0:1775764921.724492:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x30:0x0](ffff9d8184285b88) 00000080:00010000:1.0:1775764921.724494:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ea6 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.724496:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.724502:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.724517:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.724521:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=replay-single.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.724552:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.724571:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x30:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.724595:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.724602:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.724604:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ea6 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150d26 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.724621:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.724628:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c1180 x1862023065995648/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.724641:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c1180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995648:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.724675:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.725790:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c1180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995648:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.725816:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x30:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.725822:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.725843:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x30:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.725850:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x30:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.725937:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.725944:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.725946:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.725954:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.725971:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.725975:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.725980:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.725988:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995776:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.726016:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.726948:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065995776:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.726961:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.726972:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 4/1,0 mode: --/PR res: [0x200004282:0x49:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.726980:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 4/1,0 mode: --/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.726988:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.726996:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.727005:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.727013:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.727020:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.727029:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c0a80 x1862023065995776/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.727036:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.727043:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.727048:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/73 (0 0) 00000080:00200000:1.0:1775764921.727051:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.727067:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x49:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.727122:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x49:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.727129:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x49:0x0](ffff9d8184285348) for lock 0xeef44110eff3ead 00000080:00010000:1.0:1775764921.727133:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x49:0x0](ffff9d8184285348) 00000080:00010000:1.0:1775764921.727135:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ead from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.727137:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.727142:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.727153:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.727156:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.727174:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.727202:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x49:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.727226:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 1/0,0 mode: --/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.727233:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.727236:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ead lrc: 0/0,0 mode: --/PR res: [0x200004282:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150d5e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.727263:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.727267:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c0a80 x1862023065996032/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.727277:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c0a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996032:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.727307:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.728541:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c0a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996032:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.728576:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x49:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.728584:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.728618:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.728629:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x49:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.728683:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.728693:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=dne_sanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.728697:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=dne_sanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.728708:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: dne_sanity.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.728732:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.728740:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.728748:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.728760:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c2a00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996160:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.728802:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.729918:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c2a00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996160:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.729934:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.729949:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.729957:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.729965:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.729984:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.729993:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.730001:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.730008:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.730018:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81903c2a00 x1862023065996160/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.730026:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.730041:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.730050:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/6 (0 0) 00000080:00200000:1.0:1775764921.730054:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.730069:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x6:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.730138:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x6:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.730149:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x6:0x0](ffff9d81842800c8) for lock 0xeef44110eff3eb4 00000080:00010000:1.0:1775764921.730155:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x6:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.730158:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3eb4 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.730161:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.730169:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.730214:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.730221:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=dne_sanity.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.730255:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.730275:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x6:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.730302:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.730308:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.730311:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3eb4 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150d96 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.730329:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.730334:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81903c2a00 x1862023065996416/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.730348:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81903c2a00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996416:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.730382:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.731576:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81903c2a00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996416:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.731604:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.731611:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.731633:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.731641:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x6:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.731680:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.731686:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=recovery-oss-scale.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.731689:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=recovery-oss-scale.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.731697:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: recovery-oss-scale.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.731716:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.731721:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.731728:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.731739:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996544:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.731783:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.732849:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996544:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.732860:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.732873:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 4/1,0 mode: --/PR res: [0x200004282:0x29:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.732898:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 4/1,0 mode: --/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.732907:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.732915:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.732923:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.732931:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.732937:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.732945:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876fb100 x1862023065996544/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.732953:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.732962:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.732969:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/41 (0 0) 00000080:00200000:1.0:1775764921.732983:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.733003:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x29:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.733063:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x29:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.733069:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x29:0x0](ffff9d8184281988) for lock 0xeef44110eff3ebb 00000080:00010000:1.0:1775764921.733074:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x29:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.733076:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ebb from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.733079:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.733084:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.733095:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.733099:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=recovery-oss-scale.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.733117:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.733128:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x29:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.733144:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 1/0,0 mode: --/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.733149:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.733150:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ebb lrc: 0/0,0 mode: --/PR res: [0x200004282:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150dce expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.733163:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.733167:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fb100 x1862023065996800/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.733176:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996800:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.733216:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.734479:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996800:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.734514:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x29:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.734521:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.734544:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x29:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.734552:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x29:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.734602:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.734613:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=lustre-rsync-test.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.734616:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=lustre-rsync-test.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.734627:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: lustre-rsync-test.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.734651:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.734659:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.734668:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.734681:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fa680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996928:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.734714:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.735678:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fa680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065996928:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.735689:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735702:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x14:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735706:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735712:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735717:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735723:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735728:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735732:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.735739:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876fa680 x1862023065996928/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.735748:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.735758:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735766:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/20 (0 0) 00000080:00200000:1.0:1775764921.735769:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.735783:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x14:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.735842:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x14:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.735852:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x14:0x0](ffff9d8184281148) for lock 0xeef44110eff3ec2 00000080:00010000:1.0:1775764921.735858:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x14:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.735861:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ec2 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.735864:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.735869:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.735927:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.735931:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=lustre-rsync-test.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.735952:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.735963:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x14:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.735980:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.735985:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.735986:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3ec2 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150e06 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.735998:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.736002:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fa680 x1862023065997184/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.736011:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fa680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997184:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.736038:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.737138:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fa680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997184:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.737163:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.737168:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.737208:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.737215:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x14:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.737253:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.737261:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=obdfilter-survey.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.737264:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=obdfilter-survey.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.737271:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: obdfilter-survey.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.737306:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.737311:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.737316:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.737323:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997312:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.737350:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.738296:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997312:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.738306:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738326:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738331:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738337:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738342:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738348:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738353:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738357:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.738363:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f9500 x1862023065997312/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.738369:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.738375:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738380:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/26 (0 0) 00000080:00200000:1.0:1775764921.738383:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.738394:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x1a:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.738438:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x1a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.738444:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x1a:0x0](ffff9d8184285b88) for lock 0xeef44110eff3ec9 00000080:00010000:1.0:1775764921.738448:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x1a:0x0](ffff9d8184285b88) 00000080:00010000:1.0:1775764921.738450:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ec9 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.738452:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.738457:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.738469:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.738472:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=obdfilter-survey.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.738490:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.738501:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x1a:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.738523:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.738528:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.738529:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3ec9 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150e3e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.738542:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.738545:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f9500 x1862023065997568/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.738554:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997568:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.738580:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.739741:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997568:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.739767:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.739772:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.739794:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.739800:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x1a:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.739835:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.739842:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=llmountcleanup.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.739846:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=llmountcleanup.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.739856:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: llmountcleanup.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.739896:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.739903:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.739914:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.739926:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997696:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.739956:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.740921:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997696:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.740932:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.740946:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x10:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.740953:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.740962:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.740970:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.740979:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.740986:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.740992:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.741002:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f8000 x1862023065997696/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.741012:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.741021:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.741026:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/16 (0 0) 00000080:00200000:1.0:1775764921.741029:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.741041:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x10:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.741084:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x10:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.741089:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x10:0x0](ffff9d81842800c8) for lock 0xeef44110eff3ed0 00000080:00010000:1.0:1775764921.741093:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x10:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.741095:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ed0 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.741097:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.741102:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.741112:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.741116:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=llmountcleanup.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.741133:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.741144:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x10:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.741159:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.741163:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.741164:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3ed0 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150e76 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.741175:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.741179:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f8000 x1862023065997952/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.741203:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997952:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.741228:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.742398:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065997952:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.742423:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x10:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.742428:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.742457:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x10:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.742465:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x10:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.742502:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.742510:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run-llog.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.742514:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run-llog.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.742524:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: run-llog.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.742546:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.742554:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.742561:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.742573:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998080:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.742600:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.743586:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998080:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.743596:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743609:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x37:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743616:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743625:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743634:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743642:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743650:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743657:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.743665:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876fb800 x1862023065998080/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.743671:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.743678:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743683:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/55 (0 0) 00000080:00200000:1.0:1775764921.743686:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.743697:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x37:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.743737:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x37:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.743743:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x37:0x0](ffff9d8184285348) for lock 0xeef44110eff3ed7 00000080:00010000:1.0:1775764921.743747:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x37:0x0](ffff9d8184285348) 00000080:00010000:1.0:1775764921.743748:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ed7 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.743751:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.743755:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.743766:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.743769:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run-llog.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.743785:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.743796:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x37:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.743811:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.743815:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.743817:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671000/0xeef44110eff3ed7 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150eae expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.743828:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.743832:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fb800 x1862023065998336/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.743841:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998336:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.743866:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.744909:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998336:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.744933:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.744939:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.744961:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.744971:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x37:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.745019:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.745026:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=lockorder.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.745028:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=lockorder.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.745035:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: lockorder.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.745056:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.745063:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.745071:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.745083:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998464:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.745115:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.746080:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998464:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.746091:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746106:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 4/1,0 mode: --/PR res: [0x200004282:0x13:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746113:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 4/1,0 mode: --/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746121:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746130:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746138:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746145:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746152:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.746161:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f9f80 x1862023065998464/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.746171:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.746180:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746201:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/19 (0 0) 00000080:00200000:1.0:1775764921.746206:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.746222:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x13:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.746284:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x13:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.746294:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x13:0x0](ffff9d8184286c08) for lock 0xeef44110eff3ede 00000080:00010000:1.0:1775764921.746300:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x13:0x0](ffff9d8184286c08) 00000080:00010000:1.0:1775764921.746302:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ede from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.746305:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.746313:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.746329:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.746334:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=lockorder.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.746362:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.746378:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x13:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.746401:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 1/0,0 mode: --/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.746406:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.746409:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3ede lrc: 0/0,0 mode: --/PR res: [0x200004282:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150ee6 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.746424:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.746430:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f9f80 x1862023065998720/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.746443:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998720:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.746478:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.747595:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998720:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.747623:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.747629:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.747652:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.747660:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x13:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.747701:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.747709:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=recovery-scale-lib.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.747711:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=recovery-scale-lib.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.747720:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: recovery-scale-lib.sh,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.747738:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.747743:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.747748:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.747756:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fad80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998848:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.747783:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.748714:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fad80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065998848:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.748723:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748734:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748738:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748745:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748750:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748755:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748759:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748764:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.748773:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876fad80 x1862023065998848/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.748782:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.748792:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748800:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/43 (0 0) 00000080:00200000:1.0:1775764921.748804:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.748819:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x2b:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.748872:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x2b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.748894:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x2b:0x0](ffff9d8184282a08) for lock 0xeef44110eff3ee5 00000080:00010000:1.0:1775764921.748900:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x2b:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.748903:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ee5 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.748905:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.748911:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.748927:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.748932:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=recovery-scale-lib.sh, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.748951:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.748962:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x2b:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.748988:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.748992:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.748994:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ee5 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150f1e expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.749008:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.749012:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fad80 x1862023065999104/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.749021:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fad80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999104:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.749047:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.750159:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fad80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999104:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.750200:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x2b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.750211:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.750243:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x2b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.750254:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x2b:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.750309:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.750317:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=d_f4a.sanity-scrub, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.750320:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=d_f4a.sanity-scrub, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.750328:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d_f4a.sanity-scrub,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.750346:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.750354:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.750361:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.750374:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999232:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.750407:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.751265:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999232:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.751275:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751288:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 4/1,0 mode: --/PR res: [0x200004282:0x51:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751295:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 4/1,0 mode: --/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751304:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751313:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751320:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751324:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751329:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.751334:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f8380 x1862023065999232/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.751340:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751349:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/81 (0 0) 00000080:00200000:1.0:1775764921.751353:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.751368:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.751373:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200004282:0x51:0x0], updating i_size 4096 00000080:00010000:1.0:1775764921.751378:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) for lock 0xeef44110eff3eec 00000080:00010000:1.0:1775764921.751388:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.751390:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3eec from it ffffbc1987eefb20 00010000:00010000:1.0:1775764921.751393:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751401:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1775764921.751416:0:34063:0:(file.c:6226:ll_getattr_dentry()) START file d_f4a.sanity-scrub:[0x200004282:0x51:0x0](ffff9d8184281988), request_mask 2047, flags 0, foreign 0 00000002:00010000:1.0:1775764921.751427:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200004282:0x2:0x0]) in obj [0x200004282:0x2:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775764921.751431:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 30 res 8589951618/2 (0 0) 00010000:00010000:1.0:1775764921.751442:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 3/1,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751446:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 3/1,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.751450:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 3/1,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.751458:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999360:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.751482:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.752195:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999360:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.752206:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f5d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.752212:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f5d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.752218:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=4 disposition=0, status=0 req@ffff9d81876f8380 x1862023065999360/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 328/344 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1775764921.752679:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999488:192.168.202.105@tcp:37:rm.0 00000100:00100000:1.0:1775764921.752712:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.755105:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999488:192.168.202.105@tcp:37:rm.0 00010000:00010000:1.0:1775764921.755581:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f5d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.755593:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f5d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.755868:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988),name=d_f4a.sanity-scrub 00000002:00010000:1.0:1775764921.755909:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200004282:0x51:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.755923:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.755935:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.755943:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.755950:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.755954:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) for lock 0xeef44110eff3eec 00000080:00010000:1.0:1775764921.755959:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3eec from it ffffbc1987eefcb8 00010000:00010000:1.0:1775764921.755962:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.755970:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1775764921.756008:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.756013:0:34063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d_f4a.sanity-scrub, flags=258 00000080:00200000:1.0:1775764921.756026:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988), inode mode 41ed mask 44 00000080:00200008:1.0:1775764921.756033:0:34063:0:(file.c:1057:ll_file_open()) START file d_f4a.sanity-scrub:[0x200004282:0x51:0x0](ffff9d8184281988), flags 704400 00000002:00010000:1.0:1775764921.756044:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200004282:0x51:0x0]) in obj [0x200004282:0x2:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1775764921.756072:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.756079:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1775764921.756089:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1775764921.756095:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f8380 x1862023065999744/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1775764921.756104:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.756119:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999744:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.756235:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.757125:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999744:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.757134:0:34063:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757140:0:34063:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757144:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757148:0:34063:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 3/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757153:0:34063:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 4/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757156:0:34063:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190670e00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:1.0:1775764921.757159:0:34063:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 4/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757163:0:34063:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 4/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757166:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 5/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757170:0:34063:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1775764921.757174:0:34063:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 3/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.757195:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3efa lrc: 0/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.757211:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d81876f8380 x1862023065999744/t42949673055(42949673055) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1775764921.757223:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81876f8380 x1862023065999744/t42949673055(42949673055) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1775764921.757233:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 1 on i1 [0x200004282:0x2:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1775764921.757252:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200004282:0x51:0x0], updating i_size 4096 00000002:00100000:1.0:1775764921.757264:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81876f8380 x1862023065999744/t42949673055(42949673055) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1775764921.757302:0:34063:0:(file.c:6226:ll_getattr_dentry()) START file d_f4a.sanity-scrub:[0x200004282:0x51:0x0](ffff9d8184281988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1775764921.757306:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988),name=d_f4a.sanity-scrub 00000002:00010000:1.0:1775764921.757313:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200004282:0x51:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.757319:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1775764921.757325:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1775764921.757329:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.757331:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.757333:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) for lock 0xeef44110eff3eec 00000080:00010000:1.0:1775764921.757336:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3eec from it ffffbc1987eefcd8 00010000:00010000:1.0:1775764921.757338:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1775764921.757342:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1775764921.757359:0:34063:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1775764921.757364:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200004282:0x51:0x0]) in obj [0x200004282:0x51:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775764921.757368:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1775764921.757372:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000100:00100000:1.0:1775764921.757792:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999872:192.168.202.105@tcp:37:rm.0 00000100:00100000:1.0:1775764921.757827:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.760339:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023065999872:192.168.202.105@tcp:37:rm.0 00010000:00010000:1.0:1775764921.760816:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1775764921.760824:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200008:1.0:1775764921.760862:0:34063:0:(file.c:447:ll_file_release()) START file d_f4a.sanity-scrub:[0x200004282:0x51:0x0](ffff9d8184281988), flags 704000 00000002:00100000:1.0:1775764921.760906:0:34063:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81876f8380 x1862023065999744/t42949673055(42949673055) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1775764921.760913:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1775764921.760916:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fb480 x1862023066000128/t0(0) o35->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1775764921.760926:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000128:192.168.202.105@tcp:35:rm.0 00000100:00100000:1.0:1775764921.760958:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.761696:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000128:192.168.202.105@tcp:35:rm.0 00000002:00100000:1.0:1775764921.761708:0:34063:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81876f8380 x1862023065999744/t42949673055(42949673055) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1775764921.761739:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.761744:0:34063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d_f4a.sanity-scrub, flags=258 00000080:00200000:1.0:1775764921.761749:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988), inode mode 41ed mask 44 00000080:00200008:1.0:1775764921.761753:0:34063:0:(file.c:1057:ll_file_open()) START file d_f4a.sanity-scrub:[0x200004282:0x51:0x0](ffff9d8184281988), flags 704400 00000002:00010000:1.0:1775764921.761762:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200004282:0x51:0x0]) in obj [0x200004282:0x2:0x0], intent: open flags 0104240704401 00010000:00010000:1.0:1775764921.761785:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.761790:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1775764921.761796:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1775764921.761799:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fb480 x1862023066000384/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1775764921.761805:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.761812:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000384:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.761907:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.762699:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000384:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.762713:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.762726:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.762737:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 4/1,0 mode: CR/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.762746:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 4/1,0 mode: CR/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.762757:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff9d81876fb480 x1862023066000384/t42949673057(42949673057) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1775764921.762767:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81876fb480 x1862023066000384/t42949673057(42949673057) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.762776:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 3/1,0 mode: CR/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.762785:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 16 res 8589951618/81 (0 0) 00000080:00200000:1.0:1775764921.762789:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 1 on i1 [0x200004282:0x2:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1775764921.762800:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200004282:0x51:0x0], updating i_size 4096 00000080:00010000:1.0:1775764921.762804:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) for lock 0xeef44110eff3f01 00000080:00010000:1.0:1775764921.762810:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f01 from it ffffbc1987eefbd0 00010000:00010000:1.0:1775764921.762813:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 3/1,0 mode: CR/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.762820:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1775764921.762836:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81876fb480 x1862023066000384/t42949673057(42949673057) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1775764921.762895:0:34063:0:(file.c:6226:ll_getattr_dentry()) START file d_f4a.sanity-scrub:[0x200004282:0x51:0x0](ffff9d8184281988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1775764921.762901:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988),name=d_f4a.sanity-scrub 00000002:00010000:1.0:1775764921.762912:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200004282:0x51:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.762920:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1775764921.762929:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1775764921.762936:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.762940:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.762943:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) for lock 0xeef44110eff3eec 00000080:00010000:1.0:1775764921.762946:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3eec from it ffffbc1987eefcd8 00010000:00010000:1.0:1775764921.762949:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1775764921.762956:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1775764921.762977:0:34063:0:(llite_lib.c:2726:ll_statfs()) VFS Op:sb=lustre (ffff9d81892c2000) d_f4a.sanity-scrub proj=0 00000080:00000004:1.0:1775764921.762981:0:34063:0:(obd_class.h:1092:obd_statfs()) lustre-clilmv-ffff9d81892c2000: age 1744, max_age 1757 00800000:00000004:1.0:1775764921.762988:0:34063:0:(obd_class.h:1092:obd_statfs()) lustre-MDT0000-mdc-ffff9d81892c2000: age 1744, max_age 1757 00000100:00100000:1.0:1775764921.763005:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000512:192.168.202.105@tcp:41:rm.0 00000100:00100000:1.0:1775764921.763038:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.763642:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000512:192.168.202.105@tcp:41:rm.0 00800000:00000004:1.0:1775764921.763650:0:34063:0:(obd_class.h:1118:obd_statfs()) lustre-MDT0000-mdc-ffff9d81892c2000: update ffff9d819079d190 cache blocks 139402/156552 objects 199104/199812 00000080:00000004:1.0:1775764921.763653:0:34063:0:(obd_class.h:1118:obd_statfs()) lustre-clilmv-ffff9d81892c2000: update ffff9d8190799e10 cache blocks 139402/156552 objects 199104/199812 00000080:00000004:1.0:1775764921.763655:0:34063:0:(llite_lib.c:2589:ll_statfs_internal()) MDC blocks 139402/156552 objects 199104/199812 00000080:00200000:1.0:1775764921.763666:0:34063:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988) pos/size0/4096 32bit_api 0 00000080:00200000:1.0:1775764921.763684:0:34063:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:1.0:1775764921.763695:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.763704:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=f2, dir=[0x200004282:0x51:0x0](ffff9d8184281988), flags=0 00000080:00200000:1.0:1775764921.763707:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=f2, dir=[0x200004282:0x51:0x0](ffff9d8184281988), intent=getattr 00000002:00010000:1.0:1775764921.763715:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f2,[0x0:0x0:0x0]) in obj [0x200004282:0x51:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.763730:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.763735:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.763740:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.763747:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000640:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.763773:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.764757:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000640:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.764776:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.764790:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x53:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.764797:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.764806:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.764812:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.764818:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.764823:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.764827:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.764834:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f8380 x1862023066000640/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.764842:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.764851:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.764859:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/83 (0 0) 00000080:00200000:1.0:1775764921.764863:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x51:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.764904:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x53:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.764996:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x53:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.765003:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x53:0x0](ffff9d8184285b88) for lock 0xeef44110eff3f08 00000080:00010000:1.0:1775764921.765009:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x53:0x0](ffff9d8184285b88) 00000080:00010000:1.0:1775764921.765011:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f08 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.765013:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.765018:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.765029:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.765033:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=f2, dir=[0x200004282:0x51:0x0](ffff9d8184281988) 00010000:00010000:1.0:1775764921.765058:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1775764921.765071:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.765129:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.765144:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x53:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.765162:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.765166:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 1/0,0 mode: --/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1775764921.765173:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1775764921.765175:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f08 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x53:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150f9c expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.765202:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3eec lrc: 0/0,0 mode: --/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32150f56 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:1.0:1775764921.765211:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.765215:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f8380 x1862023066000896/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.765224:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000896:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.765252:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.766590:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066000896:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.766627:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x53:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.766636:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.766678:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x53:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.766690:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x53:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.766750:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.766757:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=f1, dir=[0x200004282:0x51:0x0](ffff9d8184281988), flags=0 00000080:00200000:1.0:1775764921.766760:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=f1, dir=[0x200004282:0x51:0x0](ffff9d8184281988), intent=getattr 00000002:00010000:1.0:1775764921.766772:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f1,[0x0:0x0:0x0]) in obj [0x200004282:0x51:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.766797:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.766804:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.766812:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 3/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.766825:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8e00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001024:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.766858:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.767927:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8e00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001024:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.767935:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 4/1,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.767945:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 4/1,0 mode: --/PR res: [0x200004282:0x52:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.767949:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 4/1,0 mode: --/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.767955:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.767960:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.767968:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.767975:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.767987:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.767996:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f8e00 x1862023066001024/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.768006:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.768016:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.768024:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/82 (0 0) 00000080:00200000:1.0:1775764921.768028:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x51:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.768044:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x200004282:0x52:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.768097:0:34063:0:(file.c:6766:ll_layout_conf()) [0x200004282:0x52:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.768104:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x52:0x0](ffff9d8184282a08) for lock 0xeef44110eff3f0f 00000080:00010000:1.0:1775764921.768109:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x52:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.768111:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f0f from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.768113:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.768119:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.768131:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.768135:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=f1, dir=[0x200004282:0x51:0x0](ffff9d8184281988) 00010000:00010000:1.0:1775764921.768160:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.768178:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x200004282:0x52:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.768214:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 1/0,0 mode: --/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.768219:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.768221:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f0f lrc: 0/0,0 mode: --/PR res: [0x200004282:0x52:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32150fd4 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.768233:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.768237:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f8e00 x1862023066001280/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.768246:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8e00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001280:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.768273:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.769381:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8e00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001280:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.769417:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x200004282:0x52:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.769424:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.769448:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x200004282:0x52:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.769455:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x52:0x0](ffff9d8184282a08) 00000080:00200008:1.0:1775764921.769497:0:34063:0:(file.c:447:ll_file_release()) START file d_f4a.sanity-scrub:[0x200004282:0x51:0x0](ffff9d8184281988), flags 704000 00010000:00010000:1.0:1775764921.769508:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1775764921.769527:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.769533:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=d_f4a.sanity-scrub, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), flags=0 00000080:00200000:1.0:1775764921.769535:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=d_f4a.sanity-scrub, dir=[0x200004282:0x2:0x0](ffff9d8184106c08), intent=getattr 00000002:00010000:1.0:1775764921.769553:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d_f4a.sanity-scrub,[0x0:0x0:0x0]) in obj [0x200004282:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.769571:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.769576:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.769582:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.769589:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001408:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.769615:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.770553:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001408:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.770560:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770568:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x51:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770572:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770579:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770584:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770590:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770595:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770599:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.770604:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f9c00 x1862023066001408/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.770614:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770619:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/81 (0 0) 00000080:00200000:1.0:1775764921.770621:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.770631:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200004282:0x51:0x0], updating i_size 4096 00000080:00200000:1.0:1775764921.770632:0:34063:0:(namei.c:147:ll_iget()) got inode: [0x200004282:0x51:0x0](ffff9d8184281988): rc = 0 00000080:00010000:1.0:1775764921.770635:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) for lock 0xeef44110eff3f16 00000080:00010000:1.0:1775764921.770641:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x51:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.770643:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f16 from it ffffbc1987eefe00 00010000:00010000:1.0:1775764921.770645:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.770650:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.770660:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.770664:0:34063:0:(namei.c:2305:ll_rmdir()) VFS Op:name=d_f4a.sanity-scrub, dir=[0x200004282:0x2:0x0](ffff9d8184106c08) 00010000:00010000:1.0:1775764921.770675:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa32150f5d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.770685:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.770691:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000002:00100000:1.0:1775764921.770707:0:34063:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81876fb480 x1862023066000384/t42949673057(42949673057) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1775764921.770713:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1775764921.770716:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f9c00 x1862023066001664/t0(0) o35->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1775764921.770724:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001664:192.168.202.105@tcp:35:rm.0 00000100:00100000:1.0:1775764921.770754:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.771433:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001664:192.168.202.105@tcp:35:rm.0 00000002:00100000:1.0:1775764921.771448:0:34063:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81876fb480 x1862023066000384/t42949673057(42949673057) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1775764921.771470:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.771475:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 1/0,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1775764921.771479:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32150f5d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.771486:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1775764921.771487:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f16 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x51:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215100c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.771493:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f01 lrc: 0/0,0 mode: --/CR res: [0x200004282:0x51:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32150f79 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1775764921.771500:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3ef3 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32150f5d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.771507:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.771510:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f9c00 x1862023066001920/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.771519:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001920:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.771542:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.772549:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066001920:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.772564:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x51:0x0](ffff9d8184281988) 00000080:00200008:1.0:1775764921.772590:0:34063:0:(file.c:447:ll_file_release()) START file mds1:[0x200004282:0x2:0x0](ffff9d8184106c08), flags 704000 00010000:00010000:1.0:1775764921.772604:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3cd1 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x2:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fe5b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.772620:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x1:0x0](ffff9d8184103a88), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.772630:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=mds1, dir=[0x200004282:0x1:0x0](ffff9d8184103a88), flags=0 00000080:00200000:1.0:1775764921.772633:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=mds1, dir=[0x200004282:0x1:0x0](ffff9d8184103a88), intent=getattr 00000002:00010000:1.0:1775764921.772660:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: mds1,[0x0:0x0:0x0]) in obj [0x200004282:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.772676:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.772681:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.772686:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.772695:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002048:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.772725:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.773690:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002048:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.773698:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 4/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773705:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773709:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 4/1,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773715:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773720:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773724:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773735:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773742:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.773751:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876fb480 x1862023066002048/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/688 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.773761:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773779:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/2 (0 0) 00000080:00200000:1.0:1775764921.773784:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.773797:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200004282:0x2:0x0], updating i_size 4096 00000080:00200000:1.0:1775764921.773800:0:34063:0:(namei.c:147:ll_iget()) got inode: [0x200004282:0x2:0x0](ffff9d8184106c08): rc = 0 00000080:00010000:1.0:1775764921.773804:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x2:0x0](ffff9d8184106c08) for lock 0xeef44110eff3f1d 00000080:00010000:1.0:1775764921.773810:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x2:0x0](ffff9d8184106c08) 00000080:00010000:1.0:1775764921.773813:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f1d from it ffffbc1987eefe00 00010000:00010000:1.0:1775764921.773816:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.773824:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.773836:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x1:0x0](ffff9d8184103a88), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.773840:0:34063:0:(namei.c:2305:ll_rmdir()) VFS Op:name=mds1, dir=[0x200004282:0x1:0x0](ffff9d8184103a88) 00010000:00010000:1.0:1775764921.773850:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190459600/0xeef44110eff3c45 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3214fce1 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1775764921.773858:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391a00/0xeef44110eff3cc3 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fe38 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.773868:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.773873:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3cd1 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x2:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3214fe5b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:1.0:1775764921.773906:0:34063:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d8190303b80 x1862023065969408/t42949672976(42949672976) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1775764921.773913:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1775764921.773916:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fb480 x1862023066002304/t0(0) o35->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1775764921.773925:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002304:192.168.202.105@tcp:35:rm.0 00000100:00100000:1.0:1775764921.773953:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.774674:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002304:192.168.202.105@tcp:35:rm.0 00000002:00100000:1.0:1775764921.774685:0:34063:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d8190303b80 x1862023065969408/t42949672976(42949672976) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1775764921.774712:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 1/0,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.774718:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3cd1 lrc: 1/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fe5b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.774726:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190459600/0xeef44110eff3c45 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fce1 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:1.0:1775764921.774730:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1775764921.774732:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f1d lrc: 0/0,0 mode: --/PR res: [0x200004282:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215103d expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.774737:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3cd1 lrc: 0/0,0 mode: --/CR res: [0x200004282:0x2:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fe5b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.774744:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190459600/0xeef44110eff3c45 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fce1 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:1.0:1775764921.774751:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.774754:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fb480 x1862023066002560/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.774762:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002560:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.774792:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.775859:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fb480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002560:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.775872:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x2:0x0](ffff9d8184106c08) 00000080:00200000:1.0:1775764921.775917:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x1:0x0](ffff9d8184103a88), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.775922:0:34063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=0 00000080:00200008:1.0:1775764921.775937:0:34063:0:(file.c:6226:ll_getattr_dentry()) START file mds2:[0x240003ab1:0x1:0x0](ffff9d81841042c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1775764921.775941:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8),name=mds2 00000002:00010000:1.0:1775764921.775951:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x1:0x0]) in obj [0x200004282:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.775956:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663691441/1 (0 0) 00010000:00010000:1.0:1775764921.775976:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.775981:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.775990:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.775997:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002688:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.776031:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.777050:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002688:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.777059:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 4/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777069:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777074:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777080:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777085:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777091:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777096:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777100:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.777105:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f9180 x1862023066002688/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/688 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.777111:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777116:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/1 (0 0) 00000080:00200000:1.0:1775764921.777118:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1775764921.777129:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240003ab1:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1775764921.777132:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x1:0x0](ffff9d81841042c8) for lock 0xeef44110eff3f24 00000080:00010000:1.0:1775764921.777135:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x1:0x0](ffff9d81841042c8) 00000080:00010000:1.0:1775764921.777137:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f24 from it ffffbc1987eefcb8 00010000:00010000:1.0:1775764921.777138:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777143:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.777168:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x1:0x0](ffff9d8184103a88), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.777179:0:34063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=258 00000080:00200000:1.0:1775764921.777182:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 44 00000080:00200008:1.0:1775764921.777200:0:34063:0:(file.c:1057:ll_file_open()) START file mds2:[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags 704400 00000002:00010000:1.0:1775764921.777211:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x1:0x0]) in obj [0x200004282:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1775764921.777230:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.777234:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1775764921.777239:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1775764921.777243:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f9180 x1862023066002944/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1775764921.777248:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.777256:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002944:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.777357:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.778225:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066002944:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.778232:0:34063:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 4/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778238:0:34063:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 4/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778243:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 4/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778247:0:34063:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 3/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778251:0:34063:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 4/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778256:0:34063:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190670200 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:1.0:1775764921.778258:0:34063:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 4/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778267:0:34063:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 4/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778271:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 5/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778275:0:34063:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1775764921.778278:0:34063:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 3/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.778282:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f2b lrc: 0/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.778293:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d81876f9180 x1862023066002944/t38654705669(38654705669) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/664 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1775764921.778300:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81876f9180 x1862023066002944/t38654705669(38654705669) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/664 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1775764921.778308:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 1 on i1 [0x200004282:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1775764921.778316:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240003ab1:0x1:0x0], updating i_size 4096 00000002:00100000:1.0:1775764921.778324:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81876f9180 x1862023066002944/t38654705669(38654705669) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1775764921.778360:0:34063:0:(file.c:6226:ll_getattr_dentry()) START file mds2:[0x240003ab1:0x1:0x0](ffff9d81841042c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1775764921.778364:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8),name=mds2 00000002:00010000:1.0:1775764921.778371:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x1:0x0]) in obj [0x200004282:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.778377:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.778383:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.778387:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.778390:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x1:0x0](ffff9d81841042c8) 00000080:00010000:1.0:1775764921.778392:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x1:0x0](ffff9d81841042c8) for lock 0xeef44110eff3f24 00000080:00010000:1.0:1775764921.778394:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f24 from it ffffbc1987eefcd8 00010000:00010000:1.0:1775764921.778396:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.778401:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1775764921.778415:0:34063:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1775764921.778422:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x1:0x0]) in obj [0x240003ab1:0x1:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775764921.778425:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.778430:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1775764921.778855:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fa300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003072:192.168.202.105@tcp:37:rm.0 00000100:00100000:1.0:1775764921.778922:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.781334:0:34063:0:(client.c:3082:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9d81892c2000: committing for last_committed 38654705669 gen 1 00000100:00100000:1.0:1775764921.781351:0:34063:0:(client.c:3131:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff9d81876f9180 x1862023066002944/t38654705669(38654705669) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1775764921.781362:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fa300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003072:192.168.202.105@tcp:37:rm.0 00010000:00010000:1.0:1775764921.781892:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.781905:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1775764921.781957:0:34063:0:(file.c:447:ll_file_release()) START file mds2:[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags 704000 00000002:00100000:1.0:1775764921.781991:0:34063:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81876f9180 x1862023066002944/t38654705669(38654705669) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1775764921.782001:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1775764921.782005:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fa300 x1862023066003328/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1775764921.782019:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fa300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003328:192.168.202.105@tcp:35:rm.0 00000100:00100000:1.0:1775764921.782055:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.782949:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fa300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003328:192.168.202.105@tcp:35:rm.0 00000002:00100000:1.0:1775764921.782965:0:34063:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81876f9180 x1862023066002944/t38654705669(38654705669) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1775764921.783003:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x1:0x0](ffff9d8184103a88), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.783009:0:34063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=258 00000080:00200000:1.0:1775764921.783015:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 44 00000080:00200008:1.0:1775764921.783022:0:34063:0:(file.c:1057:ll_file_open()) START file mds2:[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags 704400 00000002:00010000:1.0:1775764921.783035:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x1:0x0]) in obj [0x200004282:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:1.0:1775764921.783067:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.783076:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1775764921.783084:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1775764921.783090:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fa300 x1862023066003584/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1775764921.783098:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 3/1,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.783109:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fa300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003584:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.783228:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.784083:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fa300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003584:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.784102:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.784109:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.784116:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 4/1,0 mode: CR/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.784122:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 4/1,0 mode: CR/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.784129:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=200002b, status=0 req@ffff9d81876fa300 x1862023066003584/t38654705671(38654705671) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/664 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1775764921.784141:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81876fa300 x1862023066003584/t38654705671(38654705671) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/664 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.784148:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 3/1,0 mode: CR/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.784153:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 16 res 9663691441/1 (0 0) 00000080:00200000:1.0:1775764921.784156:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 1 on i1 [0x200004282:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1775764921.784167:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240003ab1:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1775764921.784171:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x1:0x0](ffff9d81841042c8) for lock 0xeef44110eff3f32 00000080:00010000:1.0:1775764921.784175:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f32 from it ffffbc1987eefbd0 00010000:00010000:1.0:1775764921.784182:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 3/1,0 mode: CR/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.784197:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 2/0,0 mode: CR/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1775764921.784210:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81876fa300 x1862023066003584/t38654705671(38654705671) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1775764921.784243:0:34063:0:(file.c:6226:ll_getattr_dentry()) START file mds2:[0x240003ab1:0x1:0x0](ffff9d81841042c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1775764921.784247:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8),name=mds2 00000002:00010000:1.0:1775764921.784256:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x1:0x0]) in obj [0x200004282:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.784261:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.784267:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.784275:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200004282:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.784279:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x1:0x0](ffff9d81841042c8) 00000080:00010000:1.0:1775764921.784283:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x1:0x0](ffff9d81841042c8) for lock 0xeef44110eff3f24 00000080:00010000:1.0:1775764921.784286:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f24 from it ffffbc1987eefcd8 00010000:00010000:1.0:1775764921.784289:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.784296:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1775764921.784318:0:34063:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8) pos/size0/4096 32bit_api 0 00000080:00200000:1.0:1775764921.784373:0:34063:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:1.0:1775764921.784387:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.784398:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sgpdd-survey.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.784401:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sgpdd-survey.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.784411:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sgpdd-survey.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.784433:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.784440:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.784447:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.784460:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003712:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.784495:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.785471:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003712:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.785479:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785488:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785492:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785497:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785502:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785506:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785511:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785515:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.785520:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f9180 x1862023066003712/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.785525:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.785531:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785536:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/77 (0 0) 00000080:00200000:1.0:1775764921.785538:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.785553:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x4d:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.785631:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x4d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.785637:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x4d:0x0](ffff9d8184281988) for lock 0xeef44110eff3f39 00000080:00010000:1.0:1775764921.785643:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x4d:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.785645:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f39 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.785647:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.785652:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.785663:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.785667:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sgpdd-survey.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.785686:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1775764921.785696:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459200/0xeef44110eff3c6f lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x10/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fd5f expref: -99 pid: 33385 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.785755:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.785771:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x4d:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.785789:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.785795:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1775764921.785801:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1775764921.785803:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f39 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215109f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.785815:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f24 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32151060 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00100000:1.0:1775764921.785827:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.785833:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f9180 x1862023066003968/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.785846:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003968:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.785871:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.787139:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066003968:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.787181:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.787205:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.787248:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.787261:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x4d:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.787317:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.787326:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=scrub-performance.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.787329:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=scrub-performance.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.787340:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: scrub-performance.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.787371:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.787378:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.787386:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.787398:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fbb80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004096:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.787429:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.788389:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fbb80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004096:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.788398:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788408:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788413:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788419:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788425:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788434:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788439:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788443:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.788448:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876fbb80 x1862023066004096/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.788454:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.788460:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788465:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/73 (0 0) 00000080:00200000:1.0:1775764921.788467:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.788482:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x49:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.788540:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x49:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.788551:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x49:0x0](ffff9d8184282a08) for lock 0xeef44110eff3f40 00000080:00010000:1.0:1775764921.788558:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x49:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.788561:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f40 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.788564:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.788575:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.788589:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.788593:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=scrub-performance.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.788614:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.788626:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x49:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.788642:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.788647:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.788648:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f40 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321510d7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.788660:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.788664:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876fbb80 x1862023066004352/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.788672:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876fbb80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004352:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.788698:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.789919:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876fbb80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004352:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.789959:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x49:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.789967:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.789997:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.790010:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x49:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.790065:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.790075:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-krb5.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.790078:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.790090:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-krb5.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.790113:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.790122:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.790129:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.790142:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876faa00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004480:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.790178:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.791027:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876faa00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004480:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.791037:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791049:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791056:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791063:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791077:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791085:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791091:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791097:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.791105:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876faa00 x1862023066004480/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.791117:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.791126:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791134:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/61 (0 0) 00000080:00200000:1.0:1775764921.791137:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.791153:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x3d:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.791221:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x3d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.791229:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x3d:0x0](ffff9d8184285b88) for lock 0xeef44110eff3f47 00000080:00010000:1.0:1775764921.791235:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x3d:0x0](ffff9d8184285b88) 00000080:00010000:1.0:1775764921.791239:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f47 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.791241:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.791249:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.791264:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.791268:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-krb5.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.791295:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.791312:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x3d:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.791332:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.791338:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.791349:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f47 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215110f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.791364:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.791370:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876faa00 x1862023066004736/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.791383:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876faa00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004736:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.791413:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.792624:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876faa00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004736:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.792647:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.792658:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.792678:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.792685:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x3d:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.792721:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.792734:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=parallel-scale-cifs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.792736:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=parallel-scale-cifs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.792744:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: parallel-scale-cifs.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.792760:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.792765:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.792772:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.792784:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004864:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.792814:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.793795:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066004864:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.793805:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793816:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793820:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793829:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793834:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793842:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793849:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793855:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.793864:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f9880 x1862023066004864/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.793872:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.793893:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793900:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/30 (0 0) 00000080:00200000:1.0:1775764921.793902:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.793912:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x1e:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.793953:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x1e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.793958:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x1e:0x0](ffff9d8184286c08) for lock 0xeef44110eff3f4e 00000080:00010000:1.0:1775764921.793962:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x1e:0x0](ffff9d8184286c08) 00000080:00010000:1.0:1775764921.793964:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f4e from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.793966:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.793971:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.793981:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.793984:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=parallel-scale-cifs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.794000:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.794010:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x1e:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.794027:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.794033:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.794036:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f4e lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151147 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.794051:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.794056:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f9880 x1862023066005120/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.794070:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f9880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005120:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.794102:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.795151:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f9880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005120:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.795174:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.795179:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.795212:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.795219:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x1e:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.795254:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.795260:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=obdfilter-survey.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.795262:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=obdfilter-survey.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.795269:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: obdfilter-survey.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.795285:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.795289:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.795294:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.795301:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005248:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.795328:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.796177:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005248:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.796198:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796208:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796212:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796219:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796224:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796229:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796233:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796244:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.796256:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f8a80 x1862023066005248/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.796265:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.796274:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796282:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/25 (0 0) 00000080:00200000:1.0:1775764921.796285:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.796300:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x19:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.796339:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x19:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.796345:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x19:0x0](ffff9d8184281988) for lock 0xeef44110eff3f55 00000080:00010000:1.0:1775764921.796348:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x19:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.796350:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f55 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.796352:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.796357:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.796367:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.796370:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=obdfilter-survey.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.796386:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.796399:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x19:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.796420:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.796427:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.796430:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f55 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215117f expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.796444:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.796450:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f8a80 x1862023066005504/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.796461:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005504:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.796485:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.797634:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005504:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.797658:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.797663:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.797684:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.797691:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x19:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.797724:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.797731:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-dom.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.797733:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-dom.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.797740:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-dom.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.797756:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.797762:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.797767:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.797775:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005632:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.797801:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.798649:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005632:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.798656:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798665:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798669:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798675:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798680:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798685:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798689:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798693:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.798698:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d81876f8700 x1862023066005632/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.798703:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.798708:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798712:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/58 (0 0) 00000080:00200000:1.0:1775764921.798714:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.798723:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x3a:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.798762:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x3a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.798767:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x3a:0x0](ffff9d8184285348) for lock 0xeef44110eff3f5c 00000080:00010000:1.0:1775764921.798772:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x3a:0x0](ffff9d8184285348) 00000080:00010000:1.0:1775764921.798775:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f5c from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.798778:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.798785:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.798799:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.798803:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-dom.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.798830:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.798844:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x3a:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.798865:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.798872:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.798874:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f5c lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321511b7 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.798917:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.798921:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81876f8700 x1862023066005888/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.798930:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81876f8700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005888:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.798953:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.800103:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81876f8700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066005888:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.800128:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.800134:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.800154:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.800162:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x3a:0x0](ffff9d8184285348) 00000080:00200000:1.0:1775764921.800209:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.800214:0:34063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=test-framework.sh, flags=0 00000080:00200000:1.0:1775764921.800217:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.800219:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=test-framework.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.800242:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190458c00/0xeef44110eff3c7d lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3214fd90 expref: -99 pid: 33385 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1775764921.800254:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x4e:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.800272:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190458c00/0xeef44110eff3c7d lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3214fd90 expref: -99 pid: 33385 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1775764921.800276:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.800278:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190458c00/0xeef44110eff3c7d lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3214fd90 expref: -99 pid: 33385 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000020:00100000:1.0:1775764921.800287:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.800291:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190303b80 x1862023066006144/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.800300:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190303b80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006144:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.800342:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555180 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.801456:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190303b80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006144:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.801479:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x4e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.801483:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.801500:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x4e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.801755:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x4e:0x0](ffff9d8184103248) 00000008:00010000:1.0:1775764921.801807:0:34063:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 313063 blocks 616 [cma]time: 1775764804 1775764804 0 by oinfo size 313063 blocks 616 [cma]time 1775764804 1775764804 1775764911 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000069153f75/0xeef44110eff3c84 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x1e8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa3214fd97 expref: -99 pid: 33385 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1775764921.801887:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.801895:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=replay-single-lmv.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.801898:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=replay-single-lmv.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.801906:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: replay-single-lmv.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.801927:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.801933:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.801938:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.801947:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4dc00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006272:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.801974:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.802800:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4dc00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006272:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.802808:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802817:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802825:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802831:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802836:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802840:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802844:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802848:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.802854:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8192b4dc00 x1862023066006272/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.802859:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.802864:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802869:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/46 (0 0) 00000080:00200000:1.0:1775764921.802872:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.802897:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x2e:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.802938:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x2e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.802944:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x2e:0x0](ffff9d8184282a08) for lock 0xeef44110eff3f63 00000080:00010000:1.0:1775764921.802948:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x2e:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.802950:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f63 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.802952:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.802957:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.802968:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.802971:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=replay-single-lmv.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.802996:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.803014:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x2e:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.803036:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.803041:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.803042:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670000/0xeef44110eff3f63 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215120b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.803053:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.803057:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8192b4dc00 x1862023066006528/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.803069:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4dc00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006528:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.803094:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.804086:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4dc00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006528:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.804107:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.804112:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.804130:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.804137:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x2e:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.804166:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.804171:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=parallel-scale-nfsv3.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.804173:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=parallel-scale-nfsv3.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.804180:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: parallel-scale-nfsv3.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.804207:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.804212:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.804216:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.804223:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4ca80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006656:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.804247:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.805263:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4ca80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006656:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.805272:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805285:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805291:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805299:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805306:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805322:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805330:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805338:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.805347:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8192b4ca80 x1862023066006656/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.805354:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.805359:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805365:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/32 (0 0) 00000080:00200000:1.0:1775764921.805368:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.805378:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x20:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.805417:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x20:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.805423:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x20:0x0](ffff9d81842800c8) for lock 0xeef44110eff3f6a 00000080:00010000:1.0:1775764921.805427:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x20:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.805429:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f6a from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.805431:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.805435:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.805446:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.805449:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=parallel-scale-nfsv3.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.805466:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.805477:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x20:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.805492:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.805496:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.805498:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f6a lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151243 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.805509:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.805515:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8192b4ca80 x1862023066006912/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.805528:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4ca80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006912:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.805560:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.806675:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4ca80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066006912:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.806705:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x20:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.806710:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.806730:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x20:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.806737:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x20:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.806769:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.806775:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.806778:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.806785:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lsnapshot.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.806799:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.806804:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.806809:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.806817:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4d880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007040:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.806841:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.807778:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4d880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007040:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.807786:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807797:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807802:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807807:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807812:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807817:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807821:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807824:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.807830:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8192b4d880 x1862023066007040/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.807835:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.807840:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807845:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/64 (0 0) 00000080:00200000:1.0:1775764921.807847:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.807859:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x40:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.807914:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x40:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.807920:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x40:0x0](ffff9d8184286c08) for lock 0xeef44110eff3f71 00000080:00010000:1.0:1775764921.807924:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x40:0x0](ffff9d8184286c08) 00000080:00010000:1.0:1775764921.807926:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f71 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.807928:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.807933:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.807943:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.807946:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.807967:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.807980:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x40:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.808001:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.808007:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.808010:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3f71 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215127b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.808025:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.808031:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8192b4d880 x1862023066007296/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.808044:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4d880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007296:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.808071:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.809042:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4d880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007296:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.809070:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.809076:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.809101:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.809111:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x40:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.809150:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.809156:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=parallel-scale.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.809158:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=parallel-scale.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.809165:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: parallel-scale.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.809179:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.809198:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.809206:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.809218:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4d500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007424:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.809249:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.810046:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4d500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007424:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.810055:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810067:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810071:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810076:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810081:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810085:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810089:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810094:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.810099:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8192b4d500 x1862023066007424/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.810104:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.810109:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810114:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/34 (0 0) 00000080:00200000:1.0:1775764921.810117:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.810126:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x22:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.810163:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x22:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.810168:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x22:0x0](ffff9d8184285b88) for lock 0xeef44110eff3f78 00000080:00010000:1.0:1775764921.810172:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x22:0x0](ffff9d8184285b88) 00000080:00010000:1.0:1775764921.810174:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f78 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.810176:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.810181:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.810205:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.810208:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=parallel-scale.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.810228:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.810238:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x22:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.810259:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.810266:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.810268:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3f78 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321512b3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.810284:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.810289:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8192b4d500 x1862023066007680/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.810302:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4d500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007680:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.810329:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.811301:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4d500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007680:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.811325:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x22:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.811330:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.811348:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x22:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.811355:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x22:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.811387:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.811394:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=replay-dual.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.811396:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=replay-dual.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.811402:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: replay-dual.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.811417:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.811421:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.811426:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.811433:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4f100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007808:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.811458:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.812331:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4f100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066007808:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.812340:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812355:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812361:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812369:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812377:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812384:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812390:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812397:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.812406:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8192b4f100 x1862023066007808/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.812414:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.812423:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812431:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/44 (0 0) 00000080:00200000:1.0:1775764921.812435:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.812448:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x2c:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.812494:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x2c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.812500:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x2c:0x0](ffff9d8184281148) for lock 0xeef44110eff3f7f 00000080:00010000:1.0:1775764921.812504:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x2c:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.812505:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f7f from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.812508:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.812512:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.812522:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.812525:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=replay-dual.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.812542:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.812553:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x2c:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.812568:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.812572:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.812574:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3f7f lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321512eb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.812584:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.812588:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8192b4f100 x1862023066008064/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.812597:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4f100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008064:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.812621:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.813601:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4f100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008064:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.813624:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x2c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.813628:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.813649:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x2c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.813658:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x2c:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.813702:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.813708:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=llmountcleanup.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.813710:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=llmountcleanup.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.813717:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: llmountcleanup.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.813731:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.813736:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.813740:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.813748:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4f800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008192:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.813775:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.814669:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4f800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008192:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.814679:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814695:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814700:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814705:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814710:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814715:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814720:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814724:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.814728:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8192b4f800 x1862023066008192/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.814734:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.814738:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814743:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/15 (0 0) 00000080:00200000:1.0:1775764921.814746:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.814755:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0xf:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.814802:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0xf:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.814812:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0xf:0x0](ffff9d81842821c8) for lock 0xeef44110eff3f86 00000080:00010000:1.0:1775764921.814817:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0xf:0x0](ffff9d81842821c8) 00000080:00010000:1.0:1775764921.814820:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f86 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.814823:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.814831:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.814843:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.814846:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=llmountcleanup.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.814866:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.814910:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0xf:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.814926:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.814931:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.814932:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3f86 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151323 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.814941:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.814945:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8192b4f800 x1862023066008448/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.814953:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4f800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008448:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.814977:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.815963:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4f800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008448:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.815991:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.815997:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.816023:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.816030:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0xf:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.816069:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.816075:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=dne_sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.816077:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=dne_sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.816084:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: dne_sanity.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.816098:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816103:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816107:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.816115:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4ce00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008576:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.816138:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.816914:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4ce00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008576:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.816920:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816932:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816937:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816942:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816946:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816950:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816954:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816959:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.816963:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8192b4ce00 x1862023066008576/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.816968:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.816973:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.816978:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/5 (0 0) 00000080:00200000:1.0:1775764921.816980:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.816989:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x5:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.817022:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x5:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.817028:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x5:0x0](ffff9d8184281988) for lock 0xeef44110eff3f8d 00000080:00010000:1.0:1775764921.817031:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x5:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.817033:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f8d from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.817035:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.817040:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.817049:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.817052:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=dne_sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.817068:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.817077:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x5:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.817090:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.817094:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.817096:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3f8d lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215135b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.817105:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.817109:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8192b4ce00 x1862023066008832/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.817117:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4ce00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008832:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.817141:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.818202:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4ce00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008832:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.818231:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.818237:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.818256:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.818263:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x5:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.818294:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.818299:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-lfsck.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.818301:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.818307:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lfsck.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.818321:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.818325:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.818330:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.818337:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4d180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008960:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.818361:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.819330:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4d180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066008960:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.819337:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819348:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819353:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819358:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819362:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819367:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819371:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819375:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.819380:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8192b4d180 x1862023066008960/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.819385:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.819389:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819394:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/62 (0 0) 00000080:00200000:1.0:1775764921.819397:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.819407:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x3e:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.819442:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x3e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.819447:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x3e:0x0](ffff9d81842800c8) for lock 0xeef44110eff3f94 00000080:00010000:1.0:1775764921.819451:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x3e:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.819453:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f94 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.819455:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.819459:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.819470:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.819472:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-lfsck.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.819488:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.819498:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x3e:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.819512:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.819516:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.819518:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3f94 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151393 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.819527:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.819531:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8192b4d180 x1862023066009216/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.819539:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8192b4d180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009216:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.819562:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.820551:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8192b4d180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009216:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.820579:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.820584:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.820602:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.820609:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x3e:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.820641:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.820655:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=functions.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.820658:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=functions.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.820667:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: functions.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.820703:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.820708:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.820712:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.820719:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799e300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009344:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.820743:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.821602:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799e300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009344:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.821610:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821623:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821628:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821633:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821638:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821643:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821647:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821651:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.821656:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799e300 x1862023066009344/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.821661:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.821666:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821671:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/7 (0 0) 00000080:00200000:1.0:1775764921.821674:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.821683:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x7:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.821737:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x7:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.821748:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x7:0x0](ffff9d8184282a08) for lock 0xeef44110eff3f9b 00000080:00010000:1.0:1775764921.821753:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x7:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.821756:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3f9b from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.821759:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.821768:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.821783:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.821786:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=functions.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.821807:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.821818:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x7:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.821833:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.821837:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.821839:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3f9b lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321513cb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.821849:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.821853:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799e300 x1862023066009600/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.821862:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799e300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009600:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.821902:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.822918:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799e300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009600:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.822938:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.822943:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.822961:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.822976:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x7:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.823018:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.823026:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-selinux.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.823029:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-selinux.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.823038:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-selinux.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.823055:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.823060:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.823064:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.823072:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799c000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009728:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.823095:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.823948:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799c000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009728:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.823955:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.823966:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.823971:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.823977:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.823984:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.823992:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.823998:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.824004:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.824013:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799c000 x1862023066009728/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.824022:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.824031:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.824039:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/71 (0 0) 00000080:00200000:1.0:1775764921.824043:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.824053:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x47:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.824086:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x47:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.824092:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x47:0x0](ffff9d8184281148) for lock 0xeef44110eff3fa2 00000080:00010000:1.0:1775764921.824096:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x47:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.824098:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fa2 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.824100:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.824106:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.824119:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.824124:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-selinux.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.824158:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.824176:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x47:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.824209:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.824214:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.824216:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fa2 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151403 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.824226:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.824230:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799c000 x1862023066009984/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.824238:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799c000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009984:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.824262:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.825471:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799c000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066009984:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.825506:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.825514:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.825542:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x47:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.825553:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x47:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.825593:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.825600:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=acceptance-small.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.825602:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=acceptance-small.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.825609:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: acceptance-small.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.825624:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.825630:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.825635:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.825642:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799c380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010112:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.825668:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.826765:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799c380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010112:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.826773:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826786:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826790:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826796:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826801:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826806:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826811:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826815:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.826820:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799c380 x1862023066010112/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.826826:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.826831:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826836:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/2 (0 0) 00000080:00200000:1.0:1775764921.826839:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.826850:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x2:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.826933:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.826944:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x2:0x0](ffff9d8184285b88) for lock 0xeef44110eff3fa9 00000080:00010000:1.0:1775764921.826950:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x2:0x0](ffff9d8184285b88) 00000080:00010000:1.0:1775764921.826953:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fa9 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.826956:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.826964:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.826991:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.826997:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=acceptance-small.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.827023:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.827035:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x2:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.827052:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.827056:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.827058:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fa9 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215143b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.827068:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.827072:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799c380 x1862023066010368/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.827081:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799c380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010368:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.827107:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.828333:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799c380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010368:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.828358:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.828363:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.828384:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.828391:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x2:0x0](ffff9d8184285b88) 00000080:00200000:1.0:1775764921.828435:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.828442:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=ost_oos.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.828445:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=ost_oos.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.828452:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ost_oos.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.828467:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.828472:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.828477:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.828485:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799dc00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010496:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.828510:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.829575:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799dc00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010496:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.829582:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829595:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829599:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829605:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829609:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829614:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829620:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829627:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.829636:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799dc00 x1862023066010496/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.829645:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.829654:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829663:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/28 (0 0) 00000080:00200000:1.0:1775764921.829667:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.829680:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x1c:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.829732:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x1c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.829739:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x1c:0x0](ffff9d8184281988) for lock 0xeef44110eff3fb0 00000080:00010000:1.0:1775764921.829743:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x1c:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.829744:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fb0 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.829747:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.829752:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.829762:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.829765:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=ost_oos.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.829782:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.829793:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x1c:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.829808:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.829812:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.829814:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fb0 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151473 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.829824:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.829828:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799dc00 x1862023066010752/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.829842:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799dc00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010752:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.829866:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.831097:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799dc00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010752:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.831123:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x1c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.831129:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.831151:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x1c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.831157:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x1c:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.831223:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.831233:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-compr.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.831246:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-compr.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.831254:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-compr.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.831272:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.831278:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.831283:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.831290:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ea00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010880:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.831318:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.832169:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799ea00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066010880:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.832178:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832203:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832208:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832213:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832218:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832224:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832228:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832232:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.832237:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799ea00 x1862023066010880/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.832242:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.832247:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832253:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/57 (0 0) 00000080:00200000:1.0:1775764921.832255:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.832267:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x39:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.832305:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x39:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.832311:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x39:0x0](ffff9d81842821c8) for lock 0xeef44110eff3fb7 00000080:00010000:1.0:1775764921.832320:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x39:0x0](ffff9d81842821c8) 00000080:00010000:1.0:1775764921.832321:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fb7 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.832324:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.832328:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.832339:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.832342:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-compr.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.832358:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.832368:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x39:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.832383:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.832387:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.832389:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fb7 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321514ab expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.832404:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.832408:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799ea00 x1862023066011136/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.832417:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ea00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011136:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.832440:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.833464:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799ea00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011136:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.833488:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.833493:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.833516:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.833526:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x39:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.833572:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.833579:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=rpc.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.833581:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=rpc.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.833588:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: rpc.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.833604:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.833609:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.833613:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.833621:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799e680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011264:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.833646:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.834603:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799e680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011264:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.834618:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834637:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834644:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834653:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834661:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834669:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834673:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834677:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.834683:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799e680 x1862023066011264/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.834688:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.834694:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834699:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/49 (0 0) 00000080:00200000:1.0:1775764921.834702:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.834711:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x31:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.834748:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x31:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.834754:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x31:0x0](ffff9d8184286c08) for lock 0xeef44110eff3fbe 00000080:00010000:1.0:1775764921.834758:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x31:0x0](ffff9d8184286c08) 00000080:00010000:1.0:1775764921.834760:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fbe from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.834762:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.834767:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.834776:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.834779:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=rpc.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.834801:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.834812:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x31:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.834826:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.834831:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.834833:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fbe lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321514e3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.834854:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.834860:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799e680 x1862023066011520/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.834917:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799e680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011520:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.834948:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.836098:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799e680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011520:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.836121:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x31:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.836126:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.836144:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x31:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.836150:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x31:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.836182:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.836198:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=yaml.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.836201:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=yaml.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.836207:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: yaml.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.836222:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.836227:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.836231:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.836244:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799df80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011648:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.836270:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.837178:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799df80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011648:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.837194:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837207:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837212:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837218:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837222:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837228:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837233:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837237:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.837241:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799df80 x1862023066011648/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 576/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.837247:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.837252:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837257:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/79 (0 0) 00000080:00200000:1.0:1775764921.837259:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.837270:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x4f:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.837305:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x4f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.837313:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x4f:0x0](ffff9d8184282a08) for lock 0xeef44110eff3fc5 00000080:00010000:1.0:1775764921.837319:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x4f:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.837322:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fc5 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.837325:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.837332:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.837356:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.837361:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=yaml.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.837390:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.837407:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x4f:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.837423:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.837427:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.837429:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fc5 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215151b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.837440:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.837444:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799df80 x1862023066011904/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.837453:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799df80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011904:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.837478:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.838711:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799df80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066011904:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.838750:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x4f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.838756:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.838779:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x4f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.838786:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x4f:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.838832:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.838841:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-benchmark.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.838843:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.838851:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-benchmark.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.838911:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.838916:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.838920:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.838927:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799d180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012032:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.838952:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.839902:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799d180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012032:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.839913:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.839931:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.839938:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.839953:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.839961:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.839970:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.839978:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.839985:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.839994:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799d180 x1862023066012032/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.840003:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.840011:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.840020:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/56 (0 0) 00000080:00200000:1.0:1775764921.840025:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.840039:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x38:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.840102:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x38:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.840113:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x38:0x0](ffff9d8184281148) for lock 0xeef44110eff3fcc 00000080:00010000:1.0:1775764921.840119:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x38:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.840122:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fcc from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.840125:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.840132:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.840146:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.840151:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-benchmark.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.840181:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.840214:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x38:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.840238:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.840245:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.840247:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670a00/0xeef44110eff3fcc lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151553 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.840261:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.840265:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799d180 x1862023066012288/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.840274:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799d180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012288:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.840300:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.841435:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799d180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012288:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.841460:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.841465:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.841484:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.841490:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x38:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.841534:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.841544:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup-nfs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.841547:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-nfs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.841557:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup-nfs.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.841577:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.841582:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.841587:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.841594:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799d500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012416:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.841619:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.842687:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799d500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012416:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.842698:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842716:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842721:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842727:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842732:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842737:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842742:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842747:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.842752:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799d500 x1862023066012416/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.842758:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.842763:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842768:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/76 (0 0) 00000080:00200000:1.0:1775764921.842771:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.842784:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x4c:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.842835:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x4c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.842845:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x4c:0x0](ffff9d81842800c8) for lock 0xeef44110eff3fd3 00000080:00010000:1.0:1775764921.842851:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x4c:0x0](ffff9d81842800c8) 00000080:00010000:1.0:1775764921.842853:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fd3 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.842856:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.842863:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.842925:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.842930:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=setup-nfs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.842952:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.842962:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x4c:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.842989:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.842993:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.842995:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff3fd3 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215158b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.843005:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.843009:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799d500 x1862023066012672/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.843018:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799d500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012672:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.843048:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.844249:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799d500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012672:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.844272:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x4c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.844276:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.844298:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x4c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.844308:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x4c:0x0](ffff9d81842800c8) 00000080:00200000:1.0:1775764921.844356:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.844362:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=racer.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.844365:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=racer.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.844372:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: racer.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.844387:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.844392:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.844397:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.844404:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ca80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012800:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.844430:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.845297:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799ca80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066012800:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.845307:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845324:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845332:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845341:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845348:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845354:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845358:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845362:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.845368:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799ca80 x1862023066012800/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.845373:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.845378:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845383:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/37 (0 0) 00000080:00200000:1.0:1775764921.845386:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.845394:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x25:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.845429:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x25:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.845435:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x25:0x0](ffff9d81842821c8) for lock 0xeef44110eff3fda 00000080:00010000:1.0:1775764921.845438:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x25:0x0](ffff9d81842821c8) 00000080:00010000:1.0:1775764921.845440:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fda from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.845442:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.845447:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.845457:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.845459:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=racer.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.845475:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.845486:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x25:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.845499:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.845504:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.845505:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671200/0xeef44110eff3fda lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321515c3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.845514:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.845518:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799ca80 x1862023066013056/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.845527:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ca80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013056:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.845550:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.846571:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799ca80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013056:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.846602:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.846608:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.846629:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.846636:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x25:0x0](ffff9d81842821c8) 00000080:00200000:1.0:1775764921.846667:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.846674:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=pjdfstest.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.846676:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=pjdfstest.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.846682:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: pjdfstest.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.846696:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.846701:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.846705:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.846713:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799f800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013184:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.846737:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.847601:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799f800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013184:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.847608:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847619:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847623:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847628:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847632:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847637:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847641:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847645:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.847650:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799f800 x1862023066013184/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.847656:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.847660:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847665:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/36 (0 0) 00000080:00200000:1.0:1775764921.847668:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.847676:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x24:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.847710:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x24:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.847716:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x24:0x0](ffff9d8184282a08) for lock 0xeef44110eff3fe1 00000080:00010000:1.0:1775764921.847720:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x24:0x0](ffff9d8184282a08) 00000080:00010000:1.0:1775764921.847721:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fe1 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.847723:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.847728:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.847737:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.847740:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=pjdfstest.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.847755:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.847765:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x24:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.847778:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.847782:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.847783:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670e00/0xeef44110eff3fe1 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321515fb expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.847792:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.847796:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799f800 x1862023066013440/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.847804:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799f800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013440:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.847830:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.848923:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799f800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013440:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.848944:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x24:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.848949:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.848972:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x24:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.848980:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x24:0x0](ffff9d8184282a08) 00000080:00200000:1.0:1775764921.849028:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.849036:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_dd.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.849040:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_dd.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.849049:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: run_dd.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.849063:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.849068:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.849072:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.849080:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799d880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013568:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.849104:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.849987:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799d880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013568:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.849996:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850012:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850019:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850028:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850036:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850044:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850049:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850053:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.850058:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799d880 x1862023066013568/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.850063:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.850068:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850073:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/51 (0 0) 00000080:00200000:1.0:1775764921.850076:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.850086:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x33:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.850127:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x33:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.850136:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x33:0x0](ffff9d8184286c08) for lock 0xeef44110eff3fe8 00000080:00010000:1.0:1775764921.850142:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x33:0x0](ffff9d8184286c08) 00000080:00010000:1.0:1775764921.850145:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fe8 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.850148:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.850156:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.850170:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.850175:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run_dd.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.850215:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.850226:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x33:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.850243:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.850249:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.850252:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670200/0xeef44110eff3fe8 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151633 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.850265:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.850271:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799d880 x1862023066013824/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.850283:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799d880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013824:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.850314:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.851337:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799d880 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013824:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.851362:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.851367:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.851392:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.851403:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x33:0x0](ffff9d8184286c08) 00000080:00200000:1.0:1775764921.851437:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.851444:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=maloo_upload.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.851446:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=maloo_upload.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.851453:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: maloo_upload.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.851467:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.851472:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.851477:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.851484:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799f100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013952:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.851508:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.852566:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799f100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066013952:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.852574:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852587:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852591:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852597:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852601:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852606:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852610:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852614:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.852620:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799f100 x1862023066013952/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.852630:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.852635:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852641:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/21 (0 0) 00000080:00200000:1.0:1775764921.852643:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.852653:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x15:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.852707:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x15:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.852717:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x15:0x0](ffff9d8184281988) for lock 0xeef44110eff3fef 00000080:00010000:1.0:1775764921.852723:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x15:0x0](ffff9d8184281988) 00000080:00010000:1.0:1775764921.852726:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3fef from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.852729:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.852736:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.852752:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.852757:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=maloo_upload.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.852786:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.852804:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x15:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.852829:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.852835:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.852838:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670c00/0xeef44110eff3fef lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215166b expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.852853:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.852859:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799f100 x1862023066014208/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.852872:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799f100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066014208:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.852911:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.854010:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799f100 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066014208:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.854033:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.854038:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.854056:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.854062:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x15:0x0](ffff9d8184281988) 00000080:00200000:1.0:1775764921.854095:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.854101:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_iozone.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.854103:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_iozone.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00000002:00010000:1.0:1775764921.854110:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: run_iozone.sh,[0x0:0x0:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.854125:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.854129:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.854133:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.854140:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ce00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066014336:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.854164:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555840 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.854968:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799ce00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066014336:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.854979:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855007:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855015:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 4/1,0 mode: --/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855023:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855032:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855040:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855048:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 5/2,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855055:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 4/1,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.855064:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d818799ce00 x1862023066014336/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/640 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.855072:0:34063:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1775764921.855080:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855089:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663691441/53 (0 0) 00000080:00200000:1.0:1775764921.855093:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.855105:0:34063:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240003ab1:0x35:0x0](ffff9d8184281148) 00000080:00200000:1.0:1775764921.855170:0:34063:0:(file.c:6766:ll_layout_conf()) [0x240003ab1:0x35:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1775764921.855180:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x35:0x0](ffff9d8184281148) for lock 0xeef44110eff3ff6 00000080:00010000:1.0:1775764921.855197:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x35:0x0](ffff9d8184281148) 00000080:00010000:1.0:1775764921.855200:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3ff6 from it ffffbc1987eefdf8 00010000:00010000:1.0:1775764921.855204:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.855212:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.855227:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.855231:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run_iozone.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.855260:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.855278:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x35:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.855301:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.855308:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.855310:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671800/0xeef44110eff3ff6 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321516a3 expref: -99 pid: 34063 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1775764921.855325:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.855332:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799ce00 x1862023066014592/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.855345:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ce00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066014592:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.855377:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555f00 going to sleep for 11 seconds 00000080:00200000:1.0:1775764921.928011:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240003ab1:0x50:0x0](ffff9d8184102a08),name=d_f4a.sanity-scrub 00000002:00010000:1.0:1775764921.928026:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x50:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.928035:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.928044:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.928049:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.928053:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x50:0x0](ffff9d8184102a08) 00000080:00010000:1.0:1775764921.928056:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x50:0x0](ffff9d8184102a08) for lock 0xeef44110eff40ba 00000080:00010000:1.0:1775764921.928059:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff40ba from it ffffbc1987eefcb8 00010000:00010000:1.0:1775764921.928064:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.928075:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1775764921.928113:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.928121:0:34063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d_f4a.sanity-scrub, flags=258 00000080:00200000:1.0:1775764921.928129:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x50:0x0](ffff9d8184102a08), inode mode 41ed mask 44 00000080:00200008:1.0:1775764921.928135:0:34063:0:(file.c:1057:ll_file_open()) START file d_f4a.sanity-scrub:[0x240003ab1:0x50:0x0](ffff9d8184102a08), flags 704400 00000002:00010000:1.0:1775764921.928155:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x50:0x0]) in obj [0x240003ab1:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1775764921.928200:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.928206:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 10 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1775764921.928218:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1775764921.928225:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799f480 x1862023066025856/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1775764921.928232:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 3/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 11 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.928247:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799f480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066025856:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.928364:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81915556c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.929274:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799f480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066025856:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.929282:0:34063:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 13 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929288:0:34063:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929292:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 4/1,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929296:0:34063:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 3/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929300:0:34063:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 4/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929307:0:34063:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190670800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:1.0:1775764921.929309:0:34063:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 4/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 13 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929313:0:34063:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 4/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 13 type: IBT flags: 0x206400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929317:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 5/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 13 type: IBT flags: 0x20e400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929321:0:34063:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1775764921.929324:0:34063:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 3/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 13 type: IBT flags: 0x4a0f400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.929327:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190670800/0xeef44110eff40e4 lrc: 0/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x1 rrc: 13 type: IBT flags: 0x4a0f400000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.929334:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=1 disposition=2b, status=0 req@ffff9d818799f480 x1862023066025856/t38654705729(38654705729) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1775764921.929341:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d818799f480 x1862023066025856/t38654705729(38654705729) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1775764921.929348:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 1 on i1 [0x240003ab1:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1775764921.929357:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x240003ab1:0x50:0x0], updating i_size 4096 00000002:00100000:1.0:1775764921.929367:0:34063:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d818799f480 x1862023066025856/t38654705729(38654705729) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1775764921.929403:0:34063:0:(file.c:6226:ll_getattr_dentry()) START file d_f4a.sanity-scrub:[0x240003ab1:0x50:0x0](ffff9d8184102a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1775764921.929407:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240003ab1:0x50:0x0](ffff9d8184102a08),name=d_f4a.sanity-scrub 00000002:00010000:1.0:1775764921.929414:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x50:0x0]) in obj [0x240003ab1:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.929419:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1775764921.929425:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1775764921.929430:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240003ab1:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.929432:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240003ab1:0x50:0x0](ffff9d8184102a08) 00000080:00010000:1.0:1775764921.929434:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240003ab1:0x50:0x0](ffff9d8184102a08) for lock 0xeef44110eff40ba 00000080:00010000:1.0:1775764921.929445:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff40ba from it ffffbc1987eefcd8 00010000:00010000:1.0:1775764921.929447:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00010000:00010000:1.0:1775764921.929451:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1775764921.929467:0:34063:0:(dir.c:331:ll_iterate()) VFS Op:inode=[0x240003ab1:0x50:0x0](ffff9d8184102a08) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1775764921.929473:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240003ab1:0x50:0x0]) in obj [0x240003ab1:0x50:0x0], intent: readdir flags 0 00010000:00010000:1.0:1775764921.929477:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1775764921.929481:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000100:00100000:1.0:1775764921.929924:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ed80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066026112:192.168.202.105@tcp:37:rm.0 00000100:00100000:1.0:1775764921.929955:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81915556c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.932412:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799ed80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066026112:192.168.202.105@tcp:37:rm.0 00010000:00010000:1.0:1775764921.933013:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1775764921.933023:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459400/0xeef44110eff40ba lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x50:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151cdf expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200008:1.0:1775764921.933063:0:34063:0:(file.c:447:ll_file_release()) START file d_f4a.sanity-scrub:[0x240003ab1:0x50:0x0](ffff9d8184102a08), flags 704000 00000002:00100000:1.0:1775764921.933105:0:34063:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d818799f480 x1862023066025856/t38654705729(38654705729) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 648/608 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1775764921.933117:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1775764921.933125:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799ed80 x1862023066027136/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1775764921.933142:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ed80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066027136:192.168.202.105@tcp:35:rm.0 00000100:00100000:1.0:1775764921.933201:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81915556c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.933354:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764921.933368:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066027392:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.933561:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933572:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151de9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933579:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151de9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933588:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151de9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.933613:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 1694, setting rss=1694; kms=1694, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151de9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933630:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151de9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933636:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151de9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933640:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151de9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933645:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000863f1f9f/0xeef44110eff4115 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151de9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.933655:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066026496:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.933673:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933679:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151df0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933686:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151df0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933691:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151df0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.933702:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 855, setting rss=855; kms=855, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151df0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933715:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151df0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933721:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151df0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933743:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151df0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.933765:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000600f3c79/0xeef44110eff411c lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151df0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.933800:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066026624:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.934024:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934042:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151e05 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934049:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151e05 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934069:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151e05 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.934092:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 4207, setting rss=4207; kms=4207, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151e05 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934117:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151e05 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934129:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151e05 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934134:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151e05 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934139:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2a35ce0/0xeef44110eff4131 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151e05 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.934149:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066027264:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.934166:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934172:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151e0c expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934177:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32151e0c expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934181:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32151e0c expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.934213:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 1753, setting rss=1753; kms=1753, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151e0c expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934231:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151e0c expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934239:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151e0c expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934246:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151e0c expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.934254:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000e873b69/0xeef44110eff4138 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151e0c expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.934266:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066027392:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.950182:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066030464:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.950440:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764921.950451:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066030720:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.950760:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066031104:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.950862:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.950904:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520a5 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.950934:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321520a5 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.950955:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520a5 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.950991:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 381426, setting rss=381426; kms=381426, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520a5 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951026:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520a5 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951035:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520a5 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951051:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520a5 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951068:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c4097b4a/0xeef44110eff41b6 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321520a5 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.951102:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066030464:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.951121:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066031232:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.951157:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951165:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520ac expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951171:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321520ac expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951180:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520ac expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.951210:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2475, setting rss=2475; kms=2475, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520ac expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951230:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520ac expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951235:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520ac expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951240:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520ac expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951245:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006cb5aae2/0xeef44110eff41bd lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321520ac expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.951253:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066030720:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.951265:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764921.951271:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066031360:192.168.202.105@tcp:101:ll_agl.0 00000080:00200000:1.0:1775764921.951361:0:34072:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x48:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1775764921.951369:0:34072:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240003ab1:0x48:0x0] 00000008:00010000:1.0:1775764921.951384:0:34072:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:1.0:1775764921.951388:0:34072:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81902bf4b0, osc lock ffff9d8190eee720, flags 0x400041012 00010000:00010000:1.0:1775764921.951415:0:34072:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000be48037b/0xeef44110eff41e0 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x1e5:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951424:0:34072:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000be48037b/0xeef44110eff41e0 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x1e5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:1.0:1775764921.951448:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1775764921.951454:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1775764921.951461:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81902bf4b0 (ffff9d81991c0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1775764921.951468:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1775764921.951470:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81902bf4b0 (ffff9d81991c0cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1775764921.951497:0:34072:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x48:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1775764921.951534:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951545:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520ba expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951551:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321520ba expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951558:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520ba expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.951574:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 5442, setting rss=5442; kms=5442, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520ba expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951590:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520ba expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951596:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520ba expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951601:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520ba expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951605:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000003948afc4/0xeef44110eff41cb lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321520ba expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.951611:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066031104:192.168.202.105@tcp:101:ll_agl.0 00000080:00200000:1.0:1775764921.951627:0:34072:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x2a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1775764921.951630:0:34072:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240003ab1:0x2a:0x0] 00000008:00010000:1.0:1775764921.951637:0:34072:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:1.0:1775764921.951638:0:34072:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81902bf4b0, osc lock ffff9d8190eee720, flags 0x400041012 00010000:00010000:1.0:1775764921.951652:0:34072:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b656ebc5/0xeef44110eff41e7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x1d6:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951656:0:34072:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b656ebc5/0xeef44110eff41e7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x1d6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:1.0:1775764921.951674:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1775764921.951677:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1775764921.951678:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81902bf4b0 (ffff9d81991c0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1775764921.951680:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1775764921.951682:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81902bf4b0 (ffff9d81991c0cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1775764921.951694:0:34072:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x2a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1775764921.951827:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951838:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520d6 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951848:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321520d6 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951857:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520d6 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.951888:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 16001, setting rss=16001; kms=16001, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520d6 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951904:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520d6 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951909:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520d6 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951917:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520d6 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951923:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ffb965e8/0xeef44110eff41d2 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321520d6 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.951933:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066031232:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.951948:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951963:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520dd expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951971:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321520dd expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.951982:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520dd expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.952007:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 218297, setting rss=218297; kms=218297, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520dd expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952020:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520dd expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952025:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520dd expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952047:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520dd expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952070:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f567ab2d/0xeef44110eff41d9 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321520dd expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.952093:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066031360:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.952117:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066031744:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.952383:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764921.952396:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066031872:192.168.202.105@tcp:101:ll_agl.0 00000080:00200000:1.0:1775764921.952666:0:34072:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x32:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1775764921.952677:0:34072:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240003ab1:0x32:0x0] 00000008:00010000:1.0:1775764921.952696:0:34072:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:1.0:1775764921.952700:0:34072:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81902bf4b0, osc lock ffff9d8190eee720, flags 0x400041012 00010000:00010000:1.0:1775764921.952727:0:34072:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000051f1cc24/0xeef44110eff41fc lrc: 3/1,0 mode: --/PR res: [0x280000400:0x1da:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952749:0:34072:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000051f1cc24/0xeef44110eff41fc lrc: 3/1,0 mode: --/PR res: [0x280000400:0x1da:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:1.0:1775764921.952767:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_release():245 00000020:00010000:1.0:1775764921.952774:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1775764921.952780:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81902bf4b0 (ffff9d81991c0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1775764921.952787:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8186e28000 (ffff9d81991c0cf8/0) at cl_lock_fini():72 00000020:00010000:1.0:1775764921.952792:0:34072:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81902bf4b0 (ffff9d81991c0cf8/1) at cl_lock_fini():72 00000080:00200000:1.0:1775764921.952807:0:34072:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x32:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1775764921.952866:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952894:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520f9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952905:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321520f9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952915:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321520f9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.952946:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 15476, setting rss=15476; kms=15476, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520f9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952967:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520f9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952982:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520f9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.952998:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 4/1,0 mode: PR/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321520f9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953007:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000d8fab27a/0xeef44110eff41ee lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321520f9 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.953021:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066031744:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.953042:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953051:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152100 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953058:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32152100 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953074:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152100 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.953119:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 2103, setting rss=2103; kms=2103, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152100 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953157:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152100 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953165:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152100 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953171:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152100 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953179:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004b37378b/0xeef44110eff41f5 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32152100 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.953205:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066031872:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.953277:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066032384:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.953424:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066032512:192.168.202.105@tcp:101:ll_agl.0 00010000:00010000:1.0:1775764921.953922:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953939:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152123 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953949:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32152123 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953958:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152123 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764921.953979:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 168862, setting rss=168862; kms=168862, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152123 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.953998:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152123 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.954004:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152123 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.954008:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152123 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.954013:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017025e88/0xeef44110eff4203 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32152123 expref: -99 pid: 34072 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764921.954020:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066032384:192.168.202.105@tcp:101:ll_agl.0 00000100:00100000:1.0:1775764921.954302:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904f1500 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066032256:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.954323:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.954327:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.954342:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.954348:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x1b:0x0](ffff9d8184369988) 00000080:00200000:1.0:1775764921.954381:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.954390:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=lcov-merge.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.954392:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=lcov-merge.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.954399:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459e00/0xeef44110eff40f9 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151db8 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.954406:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff40f9 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.954408:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459e00/0xeef44110eff40f9 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151db8 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.954413:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459e00/0xeef44110eff40f9 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151db8 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.954420:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.954422:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=lcov-merge.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.954439:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459e00/0xeef44110eff40f9 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151db8 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.954448:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0xd:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.954462:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459e00/0xeef44110eff40f9 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151db8 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.954466:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.954468:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459e00/0xeef44110eff40f9 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151db8 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.954476:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.954480:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799ed80 x1862023066032768/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.954489:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799ed80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066032768:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.954514:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.955556:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799ed80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066032768:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.955577:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.955581:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.955597:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.955603:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0xd:0x0](ffff9d818436aa08) 00000080:00200000:1.0:1775764921.955634:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.955640:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_IOR.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.955642:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_IOR.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.955648:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d600/0xeef44110eff4100 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151dd4 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.955655:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4100 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.955657:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d600/0xeef44110eff4100 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151dd4 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.955661:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d600/0xeef44110eff4100 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151dd4 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.955668:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.955670:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run_IOR.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.955686:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d600/0xeef44110eff4100 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151dd4 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.955695:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x34:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.955709:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d600/0xeef44110eff4100 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151dd4 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.955717:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.955718:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d600/0xeef44110eff4100 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151dd4 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.955727:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.955730:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799f480 x1862023066033024/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.955739:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799f480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066033024:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.955762:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.956706:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799f480 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066033024:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.956728:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.956732:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.956746:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.956752:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x34:0x0](ffff9d818436d348) 00000080:00200000:1.0:1775764921.956782:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.956787:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=lustre-rsync-test.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.956789:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=lustre-rsync-test.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.956795:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c200/0xeef44110eff415b lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151ef3 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.956801:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff415b from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.956803:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c200/0xeef44110eff415b lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151ef3 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.956807:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c200/0xeef44110eff415b lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151ef3 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.956815:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.956817:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=lustre-rsync-test.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.956833:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c200/0xeef44110eff415b lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151ef3 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.956842:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x13:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.956855:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c200/0xeef44110eff415b lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151ef3 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.956859:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.956861:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c200/0xeef44110eff415b lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151ef3 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.956869:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.956873:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799fb80 x1862023066033280/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.956901:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799fb80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066033280:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.956924:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.958004:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799fb80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066033280:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.958030:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.958034:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.958048:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.958054:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x13:0x0](ffff9d8184106c08) 00000080:00200000:1.0:1775764921.958082:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.958087:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanityn.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.958089:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanityn.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.958095:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d200/0xeef44110eff4162 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151f16 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.958101:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4162 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.958103:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d200/0xeef44110eff4162 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151f16 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.958107:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d200/0xeef44110eff4162 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151f16 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.958114:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.958116:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanityn.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.958131:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d200/0xeef44110eff4162 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151f16 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.958140:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x41:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.958153:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d200/0xeef44110eff4162 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151f16 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.958157:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.958159:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d200/0xeef44110eff4162 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151f16 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.958167:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.958170:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d818799c700 x1862023066033536/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.958198:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818799c700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066033536:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.958220:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.959322:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818799c700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066033536:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.959351:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x41:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.959356:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.959378:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x41:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.959385:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x41:0x0](ffff9d8184103248) 00000080:00200000:1.0:1775764921.959427:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.959435:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.959438:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.959444:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039de00/0xeef44110eff4169 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151f32 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.959451:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4169 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.959453:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039de00/0xeef44110eff4169 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151f32 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.959458:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039de00/0xeef44110eff4169 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151f32 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.959465:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.959467:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.959485:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039de00/0xeef44110eff4169 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151f32 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.959494:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x48:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.959523:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039de00/0xeef44110eff4169 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151f32 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.959535:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.959538:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039de00/0xeef44110eff4169 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151f32 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.959552:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.959557:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190552d80 x1862023066033792/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.959566:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190552d80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066033792:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.959590:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.960553:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190552d80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066033792:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.960572:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x48:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.960575:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.960589:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x48:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.960595:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x48:0x0](ffff9d8184100908) 00000080:00200000:1.0:1775764921.960622:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.960628:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=recovery-scale-lib.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.960630:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=recovery-scale-lib.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.960636:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c800/0xeef44110eff4170 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151f4e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.960641:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4170 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.960643:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c800/0xeef44110eff4170 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151f4e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.960648:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c800/0xeef44110eff4170 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151f4e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.960655:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.960657:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=recovery-scale-lib.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.960673:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c800/0xeef44110eff4170 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151f4e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.960682:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x2a:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.960696:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c800/0xeef44110eff4170 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151f4e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.960700:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.960701:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c800/0xeef44110eff4170 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151f4e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.960709:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.960713:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190553b80 x1862023066034048/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.960721:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553b80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066034048:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.960749:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.961705:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553b80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066034048:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.961731:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x2a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.961735:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.961750:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x2a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.961756:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x2a:0x0](ffff9d8184104b08) 00000080:00200000:1.0:1775764921.961786:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.961794:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=insanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.961799:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=insanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.961807:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c400/0xeef44110eff4177 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151f6a expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.961816:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4177 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.961819:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c400/0xeef44110eff4177 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151f6a expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.961826:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c400/0xeef44110eff4177 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151f6a expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.961837:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.961841:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=insanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.961870:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c400/0xeef44110eff4177 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151f6a expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.961917:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x9:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.961939:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c400/0xeef44110eff4177 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151f6a expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.961945:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.961948:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c400/0xeef44110eff4177 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151f6a expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.961960:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.961966:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190550700 x1862023066034304/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.961979:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066034304:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.962009:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.962871:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550700 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066034304:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.962909:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.962913:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.962927:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.962933:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x9:0x0](ffff9d81841021c8) 00000080:00200000:1.0:1775764921.962989:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.962994:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=lutf.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.962996:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=lutf.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.963002:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d400/0xeef44110eff417e lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151fb7 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.963008:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff417e from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.963010:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d400/0xeef44110eff417e lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151fb7 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.963014:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d400/0xeef44110eff417e lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151fb7 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.963021:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.963023:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=lutf.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.963038:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d400/0xeef44110eff417e lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151fb7 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.963047:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x14:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.963061:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d400/0xeef44110eff417e lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151fb7 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.963068:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.963070:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d400/0xeef44110eff417e lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151fb7 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.963082:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.963088:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190552300 x1862023066034560/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.963102:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190552300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066034560:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.963133:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555300 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.964040:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190552300 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066034560:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.964066:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.964071:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.964091:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.964100:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x14:0x0](ffff9d8184101148) 00000080:00200000:1.0:1775764921.964129:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.964134:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_dbench.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.964136:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_dbench.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.964142:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c600/0xeef44110eff4185 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151fef expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.964147:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4185 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.964149:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c600/0xeef44110eff4185 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151fef expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.964154:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c600/0xeef44110eff4185 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151fef expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.964161:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.964163:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run_dbench.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.964178:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c600/0xeef44110eff4185 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151fef expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.964204:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x32:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.964219:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c600/0xeef44110eff4185 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151fef expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.964224:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.964225:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c600/0xeef44110eff4185 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151fef expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.964233:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.964237:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190550000 x1862023066034816/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.964245:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066034816:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.964275:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555300 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.965251:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550000 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066034816:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.965271:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.965274:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.965287:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.965293:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x32:0x0](ffff9d8184101988) 00000080:00200000:1.0:1775764921.965320:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.965326:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-hsm.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.965328:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.965335:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ce00/0xeef44110eff418c lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152027 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.965340:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff418c from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.965342:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ce00/0xeef44110eff418c lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152027 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.965354:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ce00/0xeef44110eff418c lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32152027 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.965361:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.965363:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=sanity-hsm.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.965380:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ce00/0xeef44110eff418c lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32152027 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.965389:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x3c:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.965407:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ce00/0xeef44110eff418c lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32152027 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.965411:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.965413:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ce00/0xeef44110eff418c lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32152027 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.965421:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.965425:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190550380 x1862023066035072/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.965433:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066035072:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.965464:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555300 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.966440:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550380 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066035072:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.966463:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.966468:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.966485:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.966491:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x3c:0x0](ffff9d8184105348) 00000080:00200000:1.0:1775764921.966530:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.966539:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_tar.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.966542:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_tar.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.966552:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d000/0xeef44110eff4193 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152082 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.966562:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4193 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.966565:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d000/0xeef44110eff4193 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32152082 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.966573:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d000/0xeef44110eff4193 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32152082 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.966581:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.966584:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=run_tar.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.966604:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d000/0xeef44110eff4193 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32152082 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.966614:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x37:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.966630:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d000/0xeef44110eff4193 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32152082 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.966637:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.966639:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039d000/0xeef44110eff4193 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32152082 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.966648:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.966652:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190551c00 x1862023066035328/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.966660:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066035328:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.966686:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555300 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.967633:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551c00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066035328:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.967654:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.967658:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.967672:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.967678:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x37:0x0](ffff9d8184269988) 00000080:00200000:1.0:1775764921.967706:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.967712:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=conf-sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.967714:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=conf-sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.967720:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c000/0xeef44110eff419a lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151fb0 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.967726:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff419a from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.967728:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c000/0xeef44110eff419a lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151fb0 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.967733:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c000/0xeef44110eff419a lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151fb0 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.967740:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.967742:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=conf-sanity.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.967758:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c000/0xeef44110eff419a lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151fb0 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.967768:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x4:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.967782:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c000/0xeef44110eff419a lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151fb0 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.967786:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.967788:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039c000/0xeef44110eff419a lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151fb0 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.967796:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.967800:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190552a00 x1862023066035584/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.967809:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190552a00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066035584:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.967839:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555300 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.968816:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190552a00 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066035584:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.968836:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.968840:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.968854:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.968860:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x4:0x0](ffff9d81841000c8) 00000080:00200000:1.0:1775764921.968952:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.968958:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup-cifs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.968960:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-cifs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.968966:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039cc00/0xeef44110eff41a1 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151ff6 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.968972:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff41a1 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.968974:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039cc00/0xeef44110eff41a1 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32151ff6 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.968981:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039cc00/0xeef44110eff41a1 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32151ff6 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.968992:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.968996:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=setup-cifs.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.969024:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039cc00/0xeef44110eff41a1 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32151ff6 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.969037:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x4a:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.969052:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039cc00/0xeef44110eff41a1 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151ff6 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.969056:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.969058:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039cc00/0xeef44110eff41a1 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32151ff6 expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.969067:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.969070:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190552680 x1862023066035840/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.969079:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190552680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066035840:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.969103:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.970023:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190552680 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066035840:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.970046:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x4a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.970050:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.970064:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x4a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.970070:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x4a:0x0](ffff9d8184105b88) 00000080:00200000:1.0:1775764921.970098:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.970104:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=dom-performance.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.970106:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=dom-performance.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.970112:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ca00/0xeef44110eff41a8 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215202e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.970118:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff41a8 from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.970120:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ca00/0xeef44110eff41a8 lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215202e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.970124:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ca00/0xeef44110eff41a8 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215202e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.970131:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.970133:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=dom-performance.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.970157:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ca00/0xeef44110eff41a8 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215202e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.970173:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x6:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.970207:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ca00/0xeef44110eff41a8 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215202e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.970217:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.970218:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039ca00/0xeef44110eff41a8 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215202e expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.970228:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.970231:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190551f80 x1862023066036096/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.970240:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066036096:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.970266:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.971276:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551f80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066036096:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.971298:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.971302:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.971318:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.971323:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x6:0x0](ffff9d8184107448) 00000080:00200000:1.0:1775764921.971353:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.971358:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=conf-sanity-framework.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags=0 00000080:00200000:1.0:1775764921.971360:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=conf-sanity-framework.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8), intent=getattr 00010000:00010000:1.0:1775764921.971367:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039da00/0xeef44110eff41af lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215207b expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1775764921.971372:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff41af from it ffffbc1987eefbd8 00010000:00010000:1.0:1775764921.971375:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039da00/0xeef44110eff41af lrc: 3/1,0 mode: PR/PR res: [0x240003ab1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215207b expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.971379:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039da00/0xeef44110eff41af lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215207b expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.971394:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.971397:0:34063:0:(namei.c:2395:ll_unlink()) VFS Op:name=conf-sanity-framework.sh, dir=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00010000:00010000:1.0:1775764921.971419:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039da00/0xeef44110eff41af lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215207b expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.971430:0:34063:0:(vvp_object.c:117:vvp_conf_set()) [0x240003ab1:0x3:0x0]: losing layout lock 00010000:00010000:1.0:1775764921.971445:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039da00/0xeef44110eff41af lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215207b expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.971449:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.971451:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819039da00/0xeef44110eff41af lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215207b expref: -99 pid: 34071 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1775764921.971459:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.971463:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190551180 x1862023066036352/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.971471:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066036352:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.971494:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.972511:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551180 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066036352:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.972542:0:34063:0:(vvp_io.c:1817:vvp_io_init()) [0x240003ab1:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1775764921.972546:0:34063:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1775764921.972564:0:34063:0:(vvp_io.c:288:vvp_io_fini()) [0x240003ab1:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1775764921.972570:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x3:0x0](ffff9d81841063c8) 00000080:00200008:1.0:1775764921.972607:0:34063:0:(file.c:447:ll_file_release()) START file mds2:[0x240003ab1:0x1:0x0](ffff9d81841042c8), flags 704000 00010000:00010000:1.0:1775764921.972623:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 2/0,0 mode: CR/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1775764921.972641:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x1:0x0](ffff9d8184103a88), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.972646:0:34063:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=0 00000080:00200000:1.0:1775764921.972649:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004282:0x1:0x0](ffff9d8184103a88), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.972652:0:34063:0:(namei.c:2305:ll_rmdir()) VFS Op:name=mds2, dir=[0x200004282:0x1:0x0](ffff9d8184103a88) 00010000:00010000:1.0:1775764921.972660:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190459a00/0xeef44110eff3c68 lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3214fd51 expref: -99 pid: 33385 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.972676:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190459a00/0xeef44110eff3c68 lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fd51 expref: -99 pid: 33385 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.972681:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764921.972701:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190459a00/0xeef44110eff3c68 lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fd51 expref: -99 pid: 33385 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.972728:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 2/0,0 mode: CR/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1775764921.972764:0:34063:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81876fa300 x1862023066003584/t38654705671(38654705671) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1775764921.972775:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1775764921.972781:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190550a80 x1862023066036736/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1775764921.972796:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066036736:192.168.202.105@tcp:35:rm.0 00000100:00100000:1.0:1775764921.972826:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.973486:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066036736:192.168.202.105@tcp:35:rm.0 00000002:00100000:1.0:1775764921.973506:0:34063:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81876fa300 x1862023066003584/t38654705671(38654705671) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1775764921.973532:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459200/0xeef44110eff3c6f lrc: 2/0,0 mode: PR/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x10/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3214fd5f expref: -99 pid: 33385 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.973546:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 1/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.973550:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459200/0xeef44110eff3c6f lrc: 1/0,0 mode: --/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x10/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fd5f expref: -99 pid: 33385 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.973554:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1775764921.973555:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff3f32 lrc: 0/0,0 mode: --/CR res: [0x240003ab1:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215107c expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764921.973560:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190459200/0xeef44110eff3c6f lrc: 0/0,0 mode: --/PR res: [0x240003ab1:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fd5f expref: -99 pid: 33385 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1775764921.973568:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.973571:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190550a80 x1862023066036992/t0(0) o36->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.973579:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066036992:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.973603:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.985134:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550a80 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066036992:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.985159:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x240003ab1:0x1:0x0](ffff9d81841042c8) 00000080:00200008:1.0:1775764921.985207:0:34063:0:(file.c:447:ll_file_release()) START file d4a.sanity-scrub:[0x200004282:0x1:0x0](ffff9d8184103a88), flags 704000 00010000:00010000:1.0:1775764921.985231:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391a00/0xeef44110eff3cc3 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fe38 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1775764921.985270:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:1.0:1775764921.985282:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.985289:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1775764921.985295:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1775764921.985300:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.985304:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:1.0:1775764921.985306:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff3c37 00000080:00010000:1.0:1775764921.985311:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3c37 from it ffffbc1987eefc10 00010000:00010000:1.0:1775764921.985313:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1775764921.985319:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1775764921.985329:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:1.0:1775764921.985340:0:34063:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=d4a.sanity-scrub, dir=[0x200000007:0x1:0x0](ffff9d81842cf448), flags=0 00000080:00200000:1.0:1775764921.985343:0:34063:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=d4a.sanity-scrub, dir=[0x200000007:0x1:0x0](ffff9d81842cf448), intent=getattr 00000002:00010000:1.0:1775764921.985351:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d4a.sanity-scrub,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.985376:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.985381:0:34063:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.985391:0:34063:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34063 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1775764921.985402:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066037120:192.168.202.105@tcp:101:rm.0 00000100:00100000:1.0:1775764921.985442:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.986321:0:34063:0:(client.c:3082:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d81892c2000: committing for last_committed 42949673063 gen 1 00000100:00100000:1.0:1775764921.986328:0:34063:0:(client.c:3131:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff9d81904f0a80 x1862023065968512/t42949672973(42949672973) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1775764921.986470:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066037120:192.168.202.105@tcp:101:rm.0 00010000:00010000:1.0:1775764921.986482:0:34063:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986493:0:34063:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986501:0:34063:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 4/1,0 mode: --/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986510:0:34063:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986517:0:34063:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986524:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986532:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 5/2,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986536:0:34063:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 4/1,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1775764921.986542:0:34063:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=b, status=0 req@ffff9d8190553800 x1862023066037120/t0(0) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/688 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1775764921.986550:0:34063:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986555:0:34063:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8188126400 type 13 mode 4 res 8589951618/1 (0 0) 00000080:00200000:1.0:1775764921.986558:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1775764921.986574:0:34063:0:(llite_lib.c:2941:ll_update_inode()) inode=[0x200004282:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1775764921.986576:0:34063:0:(namei.c:147:ll_iget()) got inode: [0x200004282:0x1:0x0](ffff9d8184103a88): rc = 0 00000080:00010000:1.0:1775764921.986580:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200004282:0x1:0x0](ffff9d8184103a88) for lock 0xeef44110eff4211 00000080:00010000:1.0:1775764921.986586:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200004282:0x1:0x0](ffff9d8184103a88) 00000080:00010000:1.0:1775764921.986588:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4211 from it ffffbc1987eefe00 00010000:00010000:1.0:1775764921.986589:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 3/1,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764921.986594:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764921.986604:0:34063:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:1.0:1775764921.986609:0:34063:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1775764921.986613:0:34063:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1775764921.986618:0:34063:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1775764921.986622:0:34063:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1775764921.986624:0:34063:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:1.0:1775764921.986626:0:34063:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff3c37 00000080:00010000:1.0:1775764921.986627:0:34063:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff3c37 from it ffffbc1987eefd88 00010000:00010000:1.0:1775764921.986629:0:34063:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1775764921.986633:0:34063:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1775764921.986637:0:34063:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 3 00000080:00200000:1.0:1775764921.986640:0:34063:0:(namei.c:2305:ll_rmdir()) VFS Op:name=d4a.sanity-scrub, dir=[0x200000007:0x1:0x0](ffff9d81842cf448) 00010000:00010000:1.0:1775764921.986649:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1775764921.986695:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 2/0,0 mode: PR/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.986704:0:34063:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391a00/0xeef44110eff3cc3 lrc: 2/0,0 mode: CR/CR res: [0x200004282:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3214fe38 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:1.0:1775764921.986723:0:34063:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81904f0a80 x1862023065968512/t42949672973(42949672973) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1775764921.986734:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1775764921.986737:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190553800 x1862023066037376/t0(0) o35->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1775764921.986746:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066037376:192.168.202.105@tcp:35:rm.0 00000100:00100000:1.0:1775764921.986772:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.987434:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066037376:192.168.202.105@tcp:35:rm.0 00000002:00100000:1.0:1775764921.987445:0:34063:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81904f0a80 x1862023065968512/t42949672973(42949672973) o101->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/664 e 0 to 0 dl 1775764937 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1775764921.987470:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 1/0,0 mode: --/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.987475:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391a00/0xeef44110eff3cc3 lrc: 1/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fe38 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.987479:0:34063:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1775764921.987482:0:34063:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1775764921.987483:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190671e00/0xeef44110eff4211 lrc: 0/0,0 mode: --/PR res: [0x200004282:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa32152330 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764921.987488:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391a00/0xeef44110eff3cc3 lrc: 0/0,0 mode: --/CR res: [0x200004282:0x1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fe38 expref: -99 pid: 34063 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1775764921.987496:0:34063:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190391600/0xeef44110eff3c37 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3214fca2 expref: -99 pid: 33383 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000020:00100000:1.0:1775764921.987508:0:34063:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1775764921.987511:0:34063:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190553800 x1862023066037632/t0(0) o36->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764921.987519:0:34063:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066037632:192.168.202.105@tcp:36:rm.0 00000100:00100000:1.0:1775764921.987542:0:34063:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191555540 going to sleep for 11 seconds 00000100:00100000:1.0:1775764921.988584:0:34063:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job rm:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34063:1862023066037632:192.168.202.105@tcp:36:rm.0 00000080:00200000:1.0:1775764921.988595:0:34063:0:(llite_lib.c:1913:ll_clear_inode()) VFS Op:inode=[0x200004282:0x1:0x0](ffff9d8184103a88) 00000100:00100000:1.0:1775764922.058058:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082288512 00000100:00100000:1.0:1775764922.058063:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082288512 00000100:00100000:1.0:1775764922.058096:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 0 00000100:00100000:1.0:1775764922.058103:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:25514:x1862023082288512:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764922.058111:0:33219:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000007c53bdd4/0xeef44110eff427a lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321524b8 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.058145:0:33219:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764922.058150:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:25514:x1862023082288512:12345-192.168.202.105@tcp:104: Request processed in 48us (141us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764922.058156:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 0 00000100:00100000:1.0:1775764922.059063:0:33220:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082288640 00000100:00100000:1.0:1775764922.059067:0:33220:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082288640 00000100:00100000:1.0:1775764922.059091:0:33220:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 1 00000100:00100000:1.0:1775764922.059097:0:33220:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:19106:x1862023082288640:12345-192.168.202.105@tcp:105: 00010000:00010000:1.0:1775764922.059103:0:33220:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.059114:0:33220:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.059119:0:33220:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 5/0,1 mode: --/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.059125:0:33220:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 5/0,1 mode: PW/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.059131:0:33220:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.059139:0:33220:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 6/0,1 mode: PW/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.059161:0:33220:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d81867fd600) 00000100:00100000:1.0:1775764922.059216:0:33220:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:19106:x1862023082288640:12345-192.168.202.105@tcp:105: Request processed in 116us (222us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764922.059227:0:33220:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 1 00000100:00100000:1.0:1775764922.066383:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082288768 00000100:00100000:1.0:1775764922.066392:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082288768 00000100:00100000:1.0:1775764922.066421:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 2 00000100:00100000:1.0:1775764922.066426:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:25511:x1862023082288768:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764922.066434:0:33219:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fc930d1/0xeef44110eff42a4 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215252f expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.066468:0:33219:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764922.066473:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:25511:x1862023082288768:12345-192.168.202.105@tcp:104: Request processed in 46us (131us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764922.066478:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 2 00000100:00100000:1.0:1775764922.067405:0:33220:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082288896 00000100:00100000:1.0:1775764922.067411:0:33220:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082288896 00000100:00100000:1.0:1775764922.067442:0:33220:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 3 00000100:00100000:1.0:1775764922.067451:0:33220:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190709c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:22026:x1862023082288896:12345-192.168.202.105@tcp:105: 00010000:00010000:1.0:1775764922.067462:0:33220:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.067472:0:33220:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.067481:0:33220:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 5/0,1 mode: --/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.067490:0:33220:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.067498:0:33220:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.067507:0:33220:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.067534:0:33220:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d8190391a00) 00000100:00100000:1.0:1775764922.067564:0:33220:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190709c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:22026:x1862023082288896:12345-192.168.202.105@tcp:105: Request processed in 112us (197us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764922.067580:0:33220:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 3 00000100:00100000:1.0:1775764922.075125:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00010000:00010000:1.0:1775764922.075282:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.075287:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000001653533e/0xeef44110eff42c7 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215258a expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.075294:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190390c00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.075297:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000001653533e/0xeef44110eff42c7 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa3215258a expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.075331:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x200004a52:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.075338:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x200004a52:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.075379:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000001653533e/0xeef44110eff42c7 lrc: 2/0,0 mode: --/PR res: [0x280000401:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215258a expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.075384:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.075401:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000001653533e/0xeef44110eff42c7 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215258a expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.075406:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000001653533e/0xeef44110eff42c7 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215258a expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.075423:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.075425:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.075427:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.086113:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.094015:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00010000:00010000:1.0:1775764922.094151:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.094158:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002b6ced68/0xeef44110eff430d lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32152640 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.094169:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190391400 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.094181:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002b6ced68/0xeef44110eff430d lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32152640 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.094228:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x200004a52:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.094234:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x200004a52:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.094273:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002b6ced68/0xeef44110eff430d lrc: 2/0,0 mode: --/PR res: [0x280000401:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152640 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.094281:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.094298:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002b6ced68/0xeef44110eff430d lrc: 1/0,0 mode: --/PR res: [0x280000401:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152640 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.094306:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002b6ced68/0xeef44110eff430d lrc: 0/0,0 mode: --/PR res: [0x280000401:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152640 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.094324:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.094326:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.094329:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.102802:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.111204:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.119994:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.128033:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.135706:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.142722:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.150753:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.158745:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.159078:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8191243100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066055296:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.168386:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.176041:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8198429c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066057600:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.176672:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.184555:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.192342:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8198429880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066059904:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.193155:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.201689:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.209289:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.217516:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.226160:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.226530:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8198429180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066064512:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.228052:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8198429180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066064512:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.234906:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00010000:00010000:1.0:1775764922.235067:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.235073:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000008e2ca50d/0xeef44110eff4560 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x22c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32152c4b expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.235081:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81c63f7000 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.235085:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000008e2ca50d/0xeef44110eff4560 lrc: 3/0,0 mode: PR/PR res: [0x2c0000401:0x22c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32152c4b expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.235121:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x200004a52:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.235127:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x200004a52:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.235160:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000008e2ca50d/0xeef44110eff4560 lrc: 2/0,0 mode: --/PR res: [0x2c0000401:0x22c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152c4b expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.235166:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.235191:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000008e2ca50d/0xeef44110eff4560 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x22c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152c4b expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.235198:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000008e2ca50d/0xeef44110eff4560 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x22c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152c4b expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.235212:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.235215:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.235217:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.243280:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00010000:00010000:1.0:1775764922.243439:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.243445:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017fcc668/0xeef44110eff4583 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x22d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32152ca6 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.243453:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81867fd400 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.243457:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017fcc668/0xeef44110eff4583 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x22d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32152ca6 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.243486:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x200004a52:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.243492:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x200004a52:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.243521:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017fcc668/0xeef44110eff4583 lrc: 2/0,0 mode: --/PR res: [0x280000401:0x22d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152ca6 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.243526:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.243546:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017fcc668/0xeef44110eff4583 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x22d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152ca6 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.243554:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017fcc668/0xeef44110eff4583 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x22d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32152ca6 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.243568:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.243572:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.243575:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.244340:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.245061:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066066816:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.252956:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.259827:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.260157:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186d39180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066069120:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.260908:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.267653:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.268595:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.275498:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.276509:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.283129:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.284116:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.291627:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.292652:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.299834:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.300909:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.307937:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.308919:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.315894:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.316951:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.324734:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.325790:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.333613:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.334652:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.341979:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.342991:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.350522:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.351550:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.359717:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.360792:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.368396:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.369566:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.377138:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.378203:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.384948:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.385931:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.392219:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.393163:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.399765:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.400729:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.408447:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.409357:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.416555:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.417537:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.424869:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.425847:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.433028:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.434042:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.442030:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.442991:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.450713:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.451911:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.459592:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.460666:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.468853:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.469993:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.477720:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.478083:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d818803c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066099072:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.479666:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d818803c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066099072:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.486360:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.492795:0:10947:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:1.0:1775764922.492804:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186d1f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066101120:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.493620:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 6/2,0 mode: --/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.493632:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 6/2,0 mode: --/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32153750 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.493643:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32153750 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.493653:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32153750 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.493665:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32153750 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.493673:0:10947:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.493681:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153750 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.493687:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153750 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.493696:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153750 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.493701:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003ba771c6/0xeef44110eff499d lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x23c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32153750 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.493710:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186d1f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066101120:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764922.494974:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186d1fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066101376:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.496496:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186d1fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066101376:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.520248:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.530413:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.538760:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.545026:0:10947:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:1.0:1775764922.545038:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186d1f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066108032:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764922.547941:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.559397:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.567609:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.568665:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.579100:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.580330:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.589793:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.591055:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.599240:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.600332:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.611510:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.612640:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.622602:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.623918:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.633126:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.634352:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.645140:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.646316:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.655064:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.666683:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.676862:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.690387:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066123008:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.691066:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.691080:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32153e11 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.691091:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32153e11 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.691102:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32153e11 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.691118:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32153e11 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.691134:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.691144:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153e11 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.691155:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153e11 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.691164:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153e11 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.691188:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ba08b01/0xeef44110eff4c36 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x245:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32153e11 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.691208:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066123008:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764922.692338:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764922.692350:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066123264:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.693766:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066123264:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.700117:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.705737:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066125312:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.706459:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.706469:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32153ec7 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.706476:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32153ec7 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.706481:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32153ec7 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.706488:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32153ec7 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.706499:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.706506:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153ec7 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.706511:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153ec7 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.706515:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153ec7 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.706524:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000dd2ddf51/0xeef44110eff4c7c lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x246:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32153ec7 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.706533:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066125312:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764922.714967:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764922.715276:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066126720:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.716721:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066126720:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.720987:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066127616:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.721642:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.721651:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32153f7d expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.721658:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32153f7d expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.721664:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32153f7d expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.721671:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32153f7d expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.721682:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.721689:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153f7d expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.721694:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153f7d expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.721699:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32153f7d expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.721703:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c3e4fdd2/0xeef44110eff4cc2 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x247:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32153f7d expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.721709:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066127616:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764922.730064:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066129024:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.731412:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066129024:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.737745:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066129920:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.738477:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.738487:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 6/2,0 mode: --/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154033 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.738495:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154033 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.738501:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154033 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.738508:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 5/2,0 mode: PR/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154033 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.738513:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.738521:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 6/2,0 mode: PR/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154033 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.738526:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154033 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.738530:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 5/1,0 mode: PR/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154033 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.738535:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e013533b/0xeef44110eff4d08 lrc: 4/0,0 mode: PR/PR res: [0x2c0000401:0x248:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154033 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.738543:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066129920:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.832989:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.832998:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff4d47 lrc: 2/0,0 mode: PR/PR res: [0x200004a52:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x4dcde3aa32154144 expref: -99 pid: 34104 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.833008:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190fcb200 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:1.0:1775764922.833011:0:33224:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff4d47 lrc: 2/0,0 mode: PR/PR res: [0x200004a52:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x4dcde3aa32154144 expref: -99 pid: 34104 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.833017:0:33224:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff4d47 lrc: 2/0,0 mode: PR/PR res: [0x200004a52:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x4dcde3aa32154144 expref: -99 pid: 34104 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.833022:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff4d47 lrc: 3/0,0 mode: PR/PR res: [0x200004a52:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x4dcde3aa32154144 expref: -99 pid: 34104 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.833034:0:33224:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d81867fd800/0xeef44110eff425e lrc: 2/0,0 mode: PR/PR res: [0x200004a52:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215245d expref: -99 pid: 34089 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1775764922.833054:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff4d47 lrc: 2/0,0 mode: --/PR res: [0x200004a52:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x4dcde3aa32154144 expref: -99 pid: 34104 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.833058:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.833077:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff4d47 lrc: 1/0,0 mode: --/PR res: [0x200004a52:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x4dcde3aa32154144 expref: -99 pid: 34104 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.833081:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff4d47 lrc: 0/0,0 mode: --/PR res: [0x200004a52:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x4dcde3aa32154144 expref: -99 pid: 34104 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.833089:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.833092:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.833094:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.852066:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066135936:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.852782:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.852796:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.852807:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.852817:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.852827:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.852833:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.852841:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.852847:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.852851:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.852856:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.852865:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066135936:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.853900:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.853906:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.853914:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190fcb600 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.853919:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.853951:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.853957:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.853997:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.854002:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.854017:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.854029:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000db49be1a/0xeef44110eff4d71 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321541ec expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.854048:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.854050:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.854052:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.857139:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.857148:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190f62800/0xeef44110eff4d7f lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4dcde3aa3215420f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.857162:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190f62800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:1.0:1775764922.857167:0:33224:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190f62800/0xeef44110eff4d7f lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4dcde3aa3215420f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.857185:0:33224:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190f62800/0xeef44110eff4d7f lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4dcde3aa3215420f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.857193:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190f62800/0xeef44110eff4d7f lrc: 3/0,0 mode: PR/PR res: [0x240004281:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3215420f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.857208:0:33224:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190fca600/0xeef44110eff4d55 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154191 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.857233:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190f62800/0xeef44110eff4d7f lrc: 2/0,0 mode: --/PR res: [0x240004281:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215420f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.857237:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.857257:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190f62800/0xeef44110eff4d7f lrc: 1/0,0 mode: --/PR res: [0x240004281:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215420f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.857265:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190f62800/0xeef44110eff4d7f lrc: 0/0,0 mode: --/PR res: [0x240004281:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215420f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.857273:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.857275:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.857277:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.864072:0:10938:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00010000:00010000:1.0:1775764922.864265:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.864273:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007139cc3d/0xeef44110eff4da2 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154271 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.864283:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81902de800 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.864288:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007139cc3d/0xeef44110eff4da2 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32154271 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.864322:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.864327:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.864358:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007139cc3d/0xeef44110eff4da2 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154271 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.864365:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.864386:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007139cc3d/0xeef44110eff4da2 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154271 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.864394:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007139cc3d/0xeef44110eff4da2 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x202:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154271 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.864414:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.864417:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.864420:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.873092:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.873100:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000004a377d5/0xeef44110eff4dc5 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321542cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.873111:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d819059fa00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.873116:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000004a377d5/0xeef44110eff4dc5 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321542cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.873150:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.873157:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.873205:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000004a377d5/0xeef44110eff4dc5 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321542cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.873213:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.873235:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000004a377d5/0xeef44110eff4dc5 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321542cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.873243:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000004a377d5/0xeef44110eff4dc5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321542cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.873255:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.873258:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.873262:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.883392:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.883398:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4de8 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154327 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.883407:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d819059ec00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.883411:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4de8 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32154327 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.883434:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.883439:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.883464:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4de8 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154327 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.883469:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.883488:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4de8 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154327 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.883495:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4de8 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x203:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154327 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.883510:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.883513:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.883516:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.889093:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764922.889105:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fdca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066140928:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.889797:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.889806:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.889812:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.889817:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.889828:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.889834:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.889840:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.889845:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.889850:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.889854:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.889866:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fdca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066140928:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.890759:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.890765:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.890772:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d819059f800 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.890775:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.890801:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.890807:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.890843:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 2/0,0 mode: --/PR res: [0x280000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.890851:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.890872:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 1/0,0 mode: --/PR res: [0x280000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.890899:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e0b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154382 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.890919:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.890922:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.890924:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.898379:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.898386:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b6592748/0xeef44110eff4e2e lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321543dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.898392:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d819059ee00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.898396:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b6592748/0xeef44110eff4e2e lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321543dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.898421:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.898431:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.898470:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b6592748/0xeef44110eff4e2e lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321543dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.898477:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.898498:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b6592748/0xeef44110eff4e2e lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321543dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.898507:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b6592748/0xeef44110eff4e2e lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321543dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.898520:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.898523:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.898525:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.906515:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.906522:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e51 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154438 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.906537:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d819059f800 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.906542:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e51 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32154438 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.906570:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.906576:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.906608:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e51 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154438 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.906613:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.906629:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e51 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154438 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.906634:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000459a33db/0xeef44110eff4e51 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154438 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.906644:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.906647:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.906651:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.914953:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.914961:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4e74 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154493 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.914971:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d819059ec00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.914988:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4e74 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32154493 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.915028:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.915034:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.915074:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4e74 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154493 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.915082:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.915102:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4e74 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154493 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.915109:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000044ddd61e/0xeef44110eff4e74 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x205:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154493 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.915119:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.915120:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.915122:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.923982:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.923990:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c19317b6/0xeef44110eff4e97 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321544ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.923999:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8181372400 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.924004:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c19317b6/0xeef44110eff4e97 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321544ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.924041:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.924049:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.924094:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c19317b6/0xeef44110eff4e97 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321544ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.924102:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.924126:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c19317b6/0xeef44110eff4e97 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321544ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.924131:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c19317b6/0xeef44110eff4e97 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321544ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.924141:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.924142:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.924144:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.932802:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.932809:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000374aafa0/0xeef44110eff4eba lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154549 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.932816:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8186ccce00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.932819:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000374aafa0/0xeef44110eff4eba lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32154549 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.932847:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.932853:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.932906:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000374aafa0/0xeef44110eff4eba lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154549 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.932914:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.932937:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000374aafa0/0xeef44110eff4eba lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154549 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.932953:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000374aafa0/0xeef44110eff4eba lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x206:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32154549 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.932973:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.932977:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.932980:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.934649:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066146944:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1775764922.942321:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.942327:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006843c362/0xeef44110eff4edd lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321545a4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.942334:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8186ccda00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.942338:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006843c362/0xeef44110eff4edd lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321545a4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.942378:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.942386:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.942427:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006843c362/0xeef44110eff4edd lrc: 2/0,0 mode: --/PR res: [0x280000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321545a4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.942433:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.942449:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006843c362/0xeef44110eff4edd lrc: 1/0,0 mode: --/PR res: [0x280000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321545a4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.942458:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000006843c362/0xeef44110eff4edd lrc: 0/0,0 mode: --/PR res: [0x280000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321545a4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.942468:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.942470:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.942472:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.951300:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.951308:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bbefb14f/0xeef44110eff4f00 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321545ff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.951318:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8186cccc00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.951323:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bbefb14f/0xeef44110eff4f00 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321545ff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.951360:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.951367:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.951412:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bbefb14f/0xeef44110eff4f00 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321545ff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.951420:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.951439:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bbefb14f/0xeef44110eff4f00 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321545ff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.951448:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bbefb14f/0xeef44110eff4f00 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x207:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321545ff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.951467:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.951469:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.951471:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.952970:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066149248:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.957854:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066150144:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.958635:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.958645:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.958653:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.958664:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.958672:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.958678:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.958686:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.958695:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.958699:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.958704:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.958712:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066150144:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.959722:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775764922.959728:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.959735:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8186ccca00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:1.0:1775764922.959739:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x208:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1775764922.959776:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0xe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1775764922.959781:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0xe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1775764922.959813:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x208:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.959822:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1775764922.959842:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x208:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.959850:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000cc4ca498/0xeef44110eff4f23 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x208:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215465a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.959868:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.959870:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775764922.959871:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764922.961543:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066150400:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.966847:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764922.966861:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190645f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066151296:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.967696:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.967711:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.967722:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.967732:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.967749:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.967758:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.967772:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.967780:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.967788:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.967796:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.967813:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190645f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066151296:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764922.968971:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082315008 00000100:00100000:1.0:1775764922.968977:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082315008 00000100:00100000:1.0:1775764922.969019:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 183 00000100:00100000:1.0:1775764922.969028:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082315008:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764922.969039:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b573bb2d/0xeef44110eff4f46 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x208:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321546b5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.969098:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764922.969108:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082315008:12345-192.168.202.105@tcp:104: Request processed in 81us (200us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764922.969116:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 183 00000100:00100000:1.0:1775764922.969273:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775764922.969286:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066151552:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.970903:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066151552:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.976284:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066152448:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764922.977062:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.977076:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.977088:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.977098:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.977110:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.977125:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.977135:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.977143:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.977152:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.977160:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x209:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.977171:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066152448:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764922.978205:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764922.978214:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082315264 00000100:00100000:1.0:1775764922.978217:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082315264 00000100:00100000:1.0:1775764922.978252:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 185 00000100:00100000:1.0:1775764922.978259:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819049f850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082315264:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764922.978268:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4f69 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x209:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154710 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.978312:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764922.978319:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819049f850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082315264:12345-192.168.202.105@tcp:104: Request processed in 60us (176us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764922.978324:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 185 00000100:00100000:1.0:1775764922.978528:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066152704:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.980198:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066152704:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.987136:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082315520 00000100:00100000:1.0:1775764922.987142:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082315520 00000100:00100000:1.0:1775764922.987190:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 187 00000100:00100000:1.0:1775764922.987200:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082315520:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764922.987210:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005f1b6b85/0xeef44110eff4f8c lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x209:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215476b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.987259:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764922.987266:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082315520:12345-192.168.202.105@tcp:104: Request processed in 69us (168us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764922.987274:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 187 00000100:00100000:1.0:1775764922.988765:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190645880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066153856:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1775764922.994459:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 6/2,0 mode: --/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.994481:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 6/2,0 mode: --/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.994493:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.994509:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764922.994522:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764922.994531:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764922.994543:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.994552:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.994561:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764922.994570:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764922.994593:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066154752:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764922.995577:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082315776 00000100:00100000:1.0:1775764922.995581:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082315776 00000100:00100000:1.0:1775764922.995610:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 189 00000100:00100000:1.0:1775764922.995616:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082315776:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764922.995624:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b31f442f/0xeef44110eff4faf lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x20a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321547c6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764922.995656:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764922.995660:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082315776:12345-192.168.202.105@tcp:104: Request processed in 45us (123us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764922.995665:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 189 00000100:00100000:1.0:1775764922.995830:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066155008:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764922.997350:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066155008:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.001900:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.001912:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066155904:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.002600:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.002612:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.002630:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.002641:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.002652:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.002678:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.002687:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.002695:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.002701:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.002706:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.002715:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066155904:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.003549:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.003557:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082316032 00000100:00100000:1.0:1775764923.003560:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082316032 00000100:00100000:1.0:1775764923.003587:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 191 00000100:00100000:1.0:1775764923.003594:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796e850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34327:x1862023082316032:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.003601:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000007b5a5c23/0xeef44110eff4fd2 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x20a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154821 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.003632:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.003637:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796e850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34327:x1862023082316032:12345-192.168.202.105@tcp:104: Request processed in 45us (135us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.003642:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 191 00000100:00100000:1.0:1775764923.003795:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066156160:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.005201:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066156160:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.011635:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082316288 00000100:00100000:1.0:1775764923.011639:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082316288 00000100:00100000:1.0:1775764923.011665:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 193 00000100:00100000:1.0:1775764923.011671:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901aea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082316288:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.011678:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edb11321/0xeef44110eff4ff5 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x20b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215487c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.011707:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.011711:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901aea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082316288:12345-192.168.202.105@tcp:104: Request processed in 41us (113us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.011716:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 193 00000100:00100000:1.0:1775764923.011842:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fbad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066157312:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.013394:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fbad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066157312:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.017430:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.017439:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066158208:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.018085:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.018094:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.018103:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.018113:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.018123:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.018132:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.018142:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.018150:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.018155:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.018161:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.018194:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066158208:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.018957:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082316544 00000100:00100000:1.0:1775764923.018961:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082316544 00000100:00100000:1.0:1775764923.018986:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 195 00000100:00100000:1.0:1775764923.018994:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901ad500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082316544:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.019001:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000010d2039c/0xeef44110eff5018 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x20b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321548d7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.019031:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.019035:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ad500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082316544:12345-192.168.202.105@tcp:104: Request processed in 41us (111us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.019039:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 195 00000100:00100000:1.0:1775764923.019154:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066158464:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.020560:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066158464:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.024808:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764923.024818:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066159360:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.025600:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.025610:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.025617:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.025627:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.025634:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.025639:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.025647:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.025652:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.025657:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.025661:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.025669:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066159360:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.026596:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.026604:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082316800 00000100:00100000:1.0:1775764923.026607:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082316800 00000100:00100000:1.0:1775764923.026633:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 197 00000100:00100000:1.0:1775764923.026639:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082316800:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.026645:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b1b8511b/0xeef44110eff503b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x20c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154932 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.026674:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.026679:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082316800:12345-192.168.202.105@tcp:104: Request processed in 41us (122us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.026684:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 197 00000100:00100000:1.0:1775764923.032721:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.032731:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066160512:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.033464:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.033478:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.033499:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.033509:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.033521:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.033530:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.033542:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.033550:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.033557:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.033565:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.033582:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066160512:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.034430:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082317056 00000100:00100000:1.0:1775764923.034435:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082317056 00000100:00100000:1.0:1775764923.034463:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 199 00000100:00100000:1.0:1775764923.034468:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901aea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082317056:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.034476:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce6b4177/0xeef44110eff505e lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x20c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215498d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.034507:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.034511:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901aea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082317056:12345-192.168.202.105@tcp:104: Request processed in 44us (120us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.034515:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 199 00000100:00100000:1.0:1775764923.034651:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066160768:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.036077:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066160768:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.040383:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764923.040393:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066161664:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.041119:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.041129:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.041136:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.041141:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.041148:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.041153:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.041162:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.041172:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.041194:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.041206:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.041224:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066161664:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.042082:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082317312 00000100:00100000:1.0:1775764923.042086:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082317312 00000100:00100000:1.0:1775764923.042112:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 201 00000100:00100000:1.0:1775764923.042118:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901ad500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082317312:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.042124:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ee5c6782/0xeef44110eff5081 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x20d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321549e8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.042155:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.042159:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ad500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082317312:12345-192.168.202.105@tcp:104: Request processed in 42us (111us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.042165:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 201 00000100:00100000:1.0:1775764923.048515:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.048529:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066162816:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.049319:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.049329:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.049341:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.049350:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.049361:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.049370:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.049382:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.049388:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.049392:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.049398:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.049406:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066162816:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.050280:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.050289:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082317568 00000100:00100000:1.0:1775764923.050293:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082317568 00000100:00100000:1.0:1775764923.050319:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 203 00000100:00100000:1.0:1775764923.050324:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082317568:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.050332:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000ca3dea9/0xeef44110eff50a4 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x20d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154a43 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.050371:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.050377:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082317568:12345-192.168.202.105@tcp:104: Request processed in 52us (149us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.050386:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 203 00000100:00100000:1.0:1775764923.058790:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082317824 00000100:00100000:1.0:1775764923.058796:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082317824 00000100:00100000:1.0:1775764923.058829:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 205 00000100:00100000:1.0:1775764923.058836:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082317824:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.058844:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff50c7 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x20e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154a9e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.058893:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.058898:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082317824:12345-192.168.202.105@tcp:104: Request processed in 63us (151us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.058904:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 205 00000100:00100000:1.0:1775764923.059029:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066164224:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.060631:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066164224:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.067420:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082318080 00000100:00100000:1.0:1775764923.067426:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082318080 00000100:00100000:1.0:1775764923.067466:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 207 00000100:00100000:1.0:1775764923.067476:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082318080:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.067486:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000075d8d68b/0xeef44110eff50ea lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x20e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154af9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.067529:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.067535:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082318080:12345-192.168.202.105@tcp:104: Request processed in 62us (154us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.067542:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 207 00000100:00100000:1.0:1775764923.076034:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.076044:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082318336 00000100:00100000:1.0:1775764923.076047:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082318336 00000100:00100000:1.0:1775764923.076077:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 209 00000100:00100000:1.0:1775764923.076083:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796c050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082318336:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.076091:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000740b0f08/0xeef44110eff510d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x20f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154b54 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.076141:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.076149:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796c050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082318336:12345-192.168.202.105@tcp:104: Request processed in 65us (171us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.076156:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 209 00000100:00100000:1.0:1775764923.084360:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082318592 00000100:00100000:1.0:1775764923.084364:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082318592 00000100:00100000:1.0:1775764923.084393:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 211 00000100:00100000:1.0:1775764923.084399:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34327:x1862023082318592:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.084407:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff5130 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x20f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154baf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.084443:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.084450:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34327:x1862023082318592:12345-192.168.202.105@tcp:104: Request processed in 50us (131us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.084458:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 211 00000100:00100000:1.0:1775764923.086295:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066167680:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.092839:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066168576:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.093653:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.093668:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.093676:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.093683:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.093695:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.093705:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.093716:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.093724:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.093732:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.093740:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x210:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.093758:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066168576:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.094715:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082318848 00000100:00100000:1.0:1775764923.094720:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082318848 00000100:00100000:1.0:1775764923.094759:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 213 00000100:00100000:1.0:1775764923.094769:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082318848:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.094779:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000013a5155b/0xeef44110eff5153 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x210:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154c0a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.094827:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.094833:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082318848:12345-192.168.202.105@tcp:104: Request processed in 66us (160us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.094839:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 213 00000100:00100000:1.0:1775764923.103514:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.103523:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082319104 00000100:00100000:1.0:1775764923.103527:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082319104 00000100:00100000:1.0:1775764923.103554:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 215 00000100:00100000:1.0:1775764923.103560:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082319104:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.103568:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000740b0f08/0xeef44110eff5176 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x210:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154c65 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.103601:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.103605:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082319104:12345-192.168.202.105@tcp:104: Request processed in 46us (133us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.103610:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 215 00000100:00100000:1.0:1775764923.103750:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81904ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066169984:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.105374:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904ff480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066169984:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1775764923.111127:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.111137:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.111144:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.111150:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.111158:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.111164:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.111184:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.111190:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.111196:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.111200:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.111209:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066170880:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.112109:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082319360 00000100:00100000:1.0:1775764923.112114:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082319360 00000100:00100000:1.0:1775764923.112152:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 217 00000100:00100000:1.0:1775764923.112161:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901ae680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082319360:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.112170:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000558bc5bd/0xeef44110eff5199 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x211:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154cc0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.112228:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.112235:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ae680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082319360:12345-192.168.202.105@tcp:104: Request processed in 76us (170us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.112244:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 217 00000100:00100000:1.0:1775764923.112392:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81904fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066171136:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.113898:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066171136:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.118348:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.118357:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81904fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066172032:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.119040:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.119055:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.119062:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.119069:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.119079:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.119087:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.119097:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.119115:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.119129:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.119141:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.119156:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066172032:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.120106:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082319616 00000100:00100000:1.0:1775764923.120111:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082319616 00000100:00100000:1.0:1775764923.120143:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 219 00000100:00100000:1.0:1775764923.120149:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082319616:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.120158:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b5ea6106/0xeef44110eff51bc lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x211:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154d1b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.120208:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.120216:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082319616:12345-192.168.202.105@tcp:104: Request processed in 66us (151us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.120223:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 219 00000100:00100000:1.0:1775764923.120353:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81904fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066172288:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.121712:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066172288:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1775764923.127044:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 6/2,0 mode: --/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.127057:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 6/2,0 mode: --/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.127068:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.127078:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.127089:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.127097:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.127108:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.127116:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.127131:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.127157:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x212:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.127188:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066173184:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.128093:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.128103:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082320000 00000100:00100000:1.0:1775764923.128107:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082320000 00000100:00100000:1.0:1775764923.128133:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 221 00000100:00100000:1.0:1775764923.128139:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21000:x1862023082320000:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.128147:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff51df lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x212:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154d76 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.128187:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.128192:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21000:x1862023082320000:12345-192.168.202.105@tcp:104: Request processed in 53us (163us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.128197:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 221 00000100:00100000:1.0:1775764923.128328:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81904fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066173440:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.129770:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066173440:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.136024:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082320384 00000100:00100000:1.0:1775764923.136028:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082320384 00000100:00100000:1.0:1775764923.136056:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 223 00000100:00100000:1.0:1775764923.136063:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082320384:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.136072:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff5202 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x212:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154dd1 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.136102:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.136107:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082320384:12345-192.168.202.105@tcp:104: Request processed in 46us (136us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.136112:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 223 00000100:00100000:1.0:1775764923.137858:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066174592:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.142382:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81904fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066175488:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.143119:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.143129:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.143135:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.143141:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.143147:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.143153:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.143161:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.143167:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.143180:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.143186:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x213:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.143195:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066175488:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.144038:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082320640 00000100:00100000:1.0:1775764923.144042:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082320640 00000100:00100000:1.0:1775764923.144069:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 225 00000100:00100000:1.0:1775764923.144074:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082320640:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.144082:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d69c6400/0xeef44110eff5225 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x213:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154e2c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.144112:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.144115:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082320640:12345-192.168.202.105@tcp:104: Request processed in 42us (119us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.144120:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 225 00000100:00100000:1.0:1775764923.144281:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81904fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066175744:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.145903:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81904fe300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066175744:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.151754:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.151764:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082320896 00000100:00100000:1.0:1775764923.151766:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082320896 00000100:00100000:1.0:1775764923.151798:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 227 00000100:00100000:1.0:1775764923.151805:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796f450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082320896:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.151812:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000532ba3a8/0xeef44110eff5248 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x213:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154e87 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.151844:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.151849:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796f450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082320896:12345-192.168.202.105@tcp:104: Request processed in 46us (146us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.151854:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 227 00000100:00100000:1.0:1775764923.157920:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066177792:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.158566:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.158579:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.158589:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.158599:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.158610:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.158620:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.158630:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.158638:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.158647:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.158656:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x214:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.158675:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066177792:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.159549:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082321152 00000100:00100000:1.0:1775764923.159554:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082321152 00000100:00100000:1.0:1775764923.159583:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 229 00000100:00100000:1.0:1775764923.159589:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082321152:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.159596:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff526b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x214:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154ee2 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.159627:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.159631:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082321152:12345-192.168.202.105@tcp:104: Request processed in 44us (113us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.159636:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 229 00000100:00100000:1.0:1775764923.159802:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066178048:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.161330:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066178048:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.167441:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082321408 00000100:00100000:1.0:1775764923.167445:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082321408 00000100:00100000:1.0:1775764923.167471:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 231 00000100:00100000:1.0:1775764923.167477:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082321408:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.167485:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ab268eec/0xeef44110eff528e lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x214:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154f3d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.167523:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.167527:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082321408:12345-192.168.202.105@tcp:104: Request processed in 51us (123us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.167532:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 231 00000100:00100000:1.0:1775764923.167669:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775764923.167678:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066179200:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.169089:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066179200:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.173989:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066180096:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.174682:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.174695:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.174710:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.174718:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.174727:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.174736:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.174746:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.174754:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.174762:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.174771:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x215:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.174793:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066180096:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.175626:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.175636:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082321664 00000100:00100000:1.0:1775764923.175640:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082321664 00000100:00100000:1.0:1775764923.175672:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 233 00000100:00100000:1.0:1775764923.175680:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796d050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082321664:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.175690:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000003793b263/0xeef44110eff52b1 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x215:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154f98 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.175726:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.175730:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796d050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082321664:12345-192.168.202.105@tcp:104: Request processed in 53us (148us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.175735:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 233 00000100:00100000:1.0:1775764923.183478:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082321920 00000100:00100000:1.0:1775764923.183483:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082321920 00000100:00100000:1.0:1775764923.183525:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 235 00000100:00100000:1.0:1775764923.183533:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082321920:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.183542:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f73574c2/0xeef44110eff52d4 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x215:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32154ff3 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.183575:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.183579:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082321920:12345-192.168.202.105@tcp:104: Request processed in 49us (136us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.183584:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 235 00000100:00100000:1.0:1775764923.183727:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775764923.183736:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066181504:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.185207:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066181504:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.189749:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066182400:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.190499:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.190509:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.190517:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.190522:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.190530:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.190536:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.190544:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.190550:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.190554:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.190562:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.190570:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066182400:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.191567:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082322176 00000100:00100000:1.0:1775764923.191571:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082322176 00000100:00100000:1.0:1775764923.191600:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 237 00000100:00100000:1.0:1775764923.191605:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082322176:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.191612:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff52f7 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x216:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215504e expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.191642:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.191646:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082322176:12345-192.168.202.105@tcp:104: Request processed in 41us (115us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.191650:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 237 00000100:00100000:1.0:1775764923.191815:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066182656:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.193294:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066182656:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.198607:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.198621:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066183552:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.199325:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.199344:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.199353:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.199360:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.199368:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.199374:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.199382:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.199387:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.199391:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.199397:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.199413:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066183552:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.200378:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.200389:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082322432 00000100:00100000:1.0:1775764923.200393:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082322432 00000100:00100000:1.0:1775764923.200423:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 239 00000100:00100000:1.0:1775764923.200430:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082322432:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.200437:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff531a lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x216:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321550a9 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.200469:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.200477:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082322432:12345-192.168.202.105@tcp:104: Request processed in 45us (151us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.200482:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 239 00000100:00100000:1.0:1775764923.200607:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775764923.200617:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066183808:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.202068:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066183808:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.208284:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066184704:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.209128:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.209142:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.209158:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.209167:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.209193:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.209203:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.209213:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.209221:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.209227:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.209235:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.209248:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066184704:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.210278:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082322688 00000100:00100000:1.0:1775764923.210285:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082322688 00000100:00100000:1.0:1775764923.210334:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 241 00000100:00100000:1.0:1775764923.210342:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901af100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082322688:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.210352:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff533d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x217:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155104 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.210397:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.210405:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901af100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082322688:12345-192.168.202.105@tcp:104: Request processed in 63us (171us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.210412:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 241 00000100:00100000:1.0:1775764923.210593:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066184960:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.212380:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066184960:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.218064:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.218074:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066185856:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.218842:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.218852:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.218860:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.218870:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.218893:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.218900:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.218907:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.218913:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.218918:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.218925:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.218942:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066185856:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.219890:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082322944 00000100:00100000:1.0:1775764923.219895:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082322944 00000100:00100000:1.0:1775764923.219928:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 243 00000100:00100000:1.0:1775764923.219933:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901af800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082322944:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.219941:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000007a63e59/0xeef44110eff5360 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x217:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215515f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.219976:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.219983:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901af800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082322944:12345-192.168.202.105@tcp:104: Request processed in 48us (138us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.219990:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 243 00000100:00100000:1.0:1775764923.226711:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81902a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066187008:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.227415:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.227436:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.227446:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.227455:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.227465:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.227473:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.227484:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.227492:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.227499:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.227507:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x218:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.227525:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066187008:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.228339:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.228348:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082323200 00000100:00100000:1.0:1775764923.228351:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082323200 00000100:00100000:1.0:1775764923.228378:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 245 00000100:00100000:1.0:1775764923.228384:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796ec50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082323200:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.228391:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff5383 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x218:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321551ba expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.228419:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.228422:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796ec50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082323200:12345-192.168.202.105@tcp:104: Request processed in 41us (135us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.228427:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 245 00000100:00100000:1.0:1775764923.236017:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082323456 00000100:00100000:1.0:1775764923.236022:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082323456 00000100:00100000:1.0:1775764923.236052:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 247 00000100:00100000:1.0:1775764923.236058:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082323456:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.236066:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ab88a11b/0xeef44110eff53a6 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x218:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155215 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.236099:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.236102:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082323456:12345-192.168.202.105@tcp:104: Request processed in 46us (135us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.236107:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 247 00000100:00100000:1.0:1775764923.237712:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066188416:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1775764923.244712:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.244722:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.244730:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.244736:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.244744:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.244749:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.244757:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.244762:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.244766:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.244771:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.244777:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066189312:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.245691:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082323712 00000100:00100000:1.0:1775764923.245696:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082323712 00000100:00100000:1.0:1775764923.245728:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 249 00000100:00100000:1.0:1775764923.245736:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901af800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082323712:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.245745:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000035079dc0/0xeef44110eff53c9 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155270 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.245786:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.245801:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901af800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082323712:12345-192.168.202.105@tcp:104: Request processed in 63us (146us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.245809:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 249 00000100:00100000:1.0:1775764923.247464:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81902a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066189568:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1775764923.253249:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.253263:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.253274:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.253283:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.253294:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.253303:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.253315:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.253322:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.253335:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.253371:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.253389:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066190464:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.254271:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.254282:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082323968 00000100:00100000:1.0:1775764923.254290:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082323968 00000100:00100000:1.0:1775764923.254325:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 251 00000100:00100000:1.0:1775764923.254331:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082323968:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.254339:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ce5fa6f1/0xeef44110eff53ec lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x219:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321552cb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.254375:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.254379:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082323968:12345-192.168.202.105@tcp:104: Request processed in 50us (160us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.254385:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 251 00000100:00100000:1.0:1775764923.254532:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874e8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066190720:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.255975:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874e8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066190720:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.260461:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1775764923.260472:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066191616:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.261130:0:10944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.261149:0:10944:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.261159:0:10944:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.261166:0:10944:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.261183:0:10944:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.261189:0:10944:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.261198:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.261203:0:10944:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.261208:0:10944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.261212:0:10944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.261226:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066191616:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.262100:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082324224 00000100:00100000:1.0:1775764923.262104:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082324224 00000100:00100000:1.0:1775764923.262128:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 253 00000100:00100000:1.0:1775764923.262133:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901ae680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082324224:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.262140:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a2f09d8/0xeef44110eff540f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155326 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.262181:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.262187:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ae680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082324224:12345-192.168.202.105@tcp:104: Request processed in 54us (159us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.262196:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 253 00000100:00100000:1.0:1775764923.269868:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082324480 00000100:00100000:1.0:1775764923.269872:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082324480 00000100:00100000:1.0:1775764923.269911:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 255 00000100:00100000:1.0:1775764923.269916:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082324480:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.269924:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000072672f45/0xeef44110eff5432 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x21a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155381 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.269954:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.269958:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082324480:12345-192.168.202.105@tcp:104: Request processed in 43us (127us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.269963:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 255 00000100:00100000:1.0:1775764923.271537:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066193024:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.275956:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066193920:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.276681:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.276694:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.276703:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.276712:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.276724:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.276731:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.276743:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.276751:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.276758:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.276772:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.276794:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066193920:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.277688:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.277698:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082324736 00000100:00100000:1.0:1775764923.277702:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082324736 00000100:00100000:1.0:1775764923.277737:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 257 00000100:00100000:1.0:1775764923.277743:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082324736:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.277750:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff5455 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x21b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321553dc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.277785:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.277791:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082324736:12345-192.168.202.105@tcp:104: Request processed in 47us (154us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.277798:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 257 00000100:00100000:1.0:1775764923.277971:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066194176:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.279420:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066194176:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.285873:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082324992 00000100:00100000:1.0:1775764923.285894:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082324992 00000100:00100000:1.0:1775764923.285926:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 259 00000100:00100000:1.0:1775764923.285933:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082324992:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.285941:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000f73574c2/0xeef44110eff5478 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x21b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155437 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.285974:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.285978:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082324992:12345-192.168.202.105@tcp:104: Request processed in 47us (181us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.285984:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 259 00000100:00100000:1.0:1775764923.287522:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874ea300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066195328:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.291726:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874ebb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066196224:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.292398:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.292408:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.292416:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.292430:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.292441:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.292449:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.292459:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.292467:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.292475:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.292481:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.292497:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874ebb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066196224:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.293363:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082325248 00000100:00100000:1.0:1775764923.293368:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082325248 00000100:00100000:1.0:1775764923.293401:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 261 00000100:00100000:1.0:1775764923.293411:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082325248:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.293420:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff549b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155492 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.293462:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.293469:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082325248:12345-192.168.202.105@tcp:104: Request processed in 60us (145us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.293476:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 261 00000100:00100000:1.0:1775764923.293622:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066196480:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1775764923.294981:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066196480:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.296751:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.296767:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.296772:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.296776:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.296781:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.296784:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.296786:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.296788:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.296790:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.296792:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.296796:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.296801:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.296803:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.296805:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.296807:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.296808:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.296813:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=run_tar.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.296815:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_tar.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.296819:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: run_tar.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.296829:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.296836:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.296840:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.296847:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066196992:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.296870:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9780 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.297647:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066196992:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.297653:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297657:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297661:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297664:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297668:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297671:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d9000 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.297672:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297675:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297679:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297682:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.297685:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297688:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9000/0xeef44110eff54b0 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.297694:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81874eb800 x1862023066196992/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.297699:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.297706:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.297734:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.297738:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.297742:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.297748:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.297752:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.297754:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.297755:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.297757:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.297759:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.297767:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.297772:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.297774:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.297776:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.297777:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.297779:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.297784:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.297786:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=run_tar.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da000, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.297789:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=run_tar.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.297797:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: run_tar.sh,[0x240004281:0x37:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.297811:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.297815:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.297820:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.297823:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81874eb800 x1862023066197248/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.297828:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.297835:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066197248:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.297912:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9780 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.298938:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066197248:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.298949:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.298971:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x37:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.298976:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.298981:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.298986:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.298992:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.298997:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.299001:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.299006:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81874eb800 x1862023066197248/t38654705867(38654705867) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.299012:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81874eb800 x1862023066197248/t38654705867(38654705867) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.299019:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.299025:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.299029:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/55 (0 0) 00000080:00200000:2.0:1775764923.299032:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.299036:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=run_tar.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.299047:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x37:0x0](ffff9d81843f63c8) 00000080:00200000:2.0:1775764923.299106:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x37:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.299115:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x37:0x0](ffff9d81843f63c8) 00000080:00010000:2.0:1775764923.299118:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x37:0x0](ffff9d81843f63c8) for lock 0xeef44110eff54b7 00000080:00010000:2.0:1775764923.299124:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x37:0x0](ffff9d81843f63c8) 00000080:00200008:2.0:1775764923.299130:0:34105:0:(file.c:1057:ll_file_open()) START file run_tar.sh:[0x240004281:0x37:0x0](ffff9d81843f63c8), flags 100301 00000002:00100000:2.0:1775764923.299143:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81874eb800 x1862023066197248/t38654705867(38654705867) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.299188:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54b7 from it ffff9d8190e6e420 00010000:00010000:2.0:1775764923.299192:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.299200:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.299210:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x37:0x0](ffff9d81843f63c8), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.299218:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file run_tar.sh:[0x240004281:0x37:0x0](ffff9d81843f63c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.299220:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x37:0x0](ffff9d81843f63c8),name=run_tar.sh 00000002:00010000:2.0:1775764923.299227:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x37:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.299232:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.299237:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.299241:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.299243:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x37:0x0](ffff9d81843f63c8) 00000080:00010000:2.0:1775764923.299245:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x37:0x0](ffff9d81843f63c8) for lock 0xeef44110eff54b7 00000080:00010000:2.0:1775764923.299247:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54b7 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.299249:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.299253:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.299265:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x37:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.299269:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x37:0x0] 00000008:00010000:2.0:1775764923.299284:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.299286:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420ab0, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.299302:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff54be lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.299308:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff54be lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.299328:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1775764923.300127:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x37:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.300132:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.300135:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.300139:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420ab0 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.300145:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.300147:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420ab0 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.300161:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x37:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.300194:0:34105:0:(file.c:2612:do_file_write_iter()) START file run_tar.sh:[0x240004281:0x37:0x0], ppos: 0, count: 1654 00000080:00200000:2.0:1775764923.300200:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 1654 bytes 00000080:00200000:2.0:1775764923.300203:0:34105:0:(file.c:2588:ll_do_tiny_write()) result: 0, original count 1654 00000080:00200000:2.0:1775764923.300211:0:34105:0:(file.c:2058:ll_file_io_generic()) run_tar.sh: write ppos: 0, bytes: 1654 00000080:00200000:2.0:1775764923.300214:0:34105:0:(file.c:1815:ll_io_set_mirror()) run_tar.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.300217:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000484a383b[0x110, 1, [0x240004281:0x37:0x0] hash] 00000020:00200000:2.0:1775764923.300218:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1654) 0 0 00000080:00200000:2.0:1775764923.300220:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x37:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.300224:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1775764923.300228:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1654) 1654, 1654 00020000:00200000:2.0:1775764923.300231:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.300241:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1775764923.300245:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.300247:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420ab0, osc lock ffff9d8198381db0, flags 0x0 00010000:00010000:2.0:1775764923.300252:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/540 (0 4095) 00010000:00010000:2.0:1775764923.300267:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.300275:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.300288:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.300301:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874e9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066197504:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.300332:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9900 going to sleep for 11 seconds 00010000:00010000:2.0:1775764923.301129:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.301136:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff54be lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321554ed expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.301145:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8600 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.301148:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff54be lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321554ed expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.301185:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.301190:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.301219:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff54be lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321554ed expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.301227:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.301248:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff54be lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321554ed expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.301256:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff54be lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321554ed expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.301276:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.301279:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.301281:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.301696:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874e9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066197504:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.301704:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.301710:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.301714:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.302094:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.302106:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.302114:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.302122:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.302132:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.302137:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.302152:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) run_tar.sh: write [0, 1654) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.302158:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 1654 bytes 00000080:00200000:2.0:1775764923.302192:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 1654, copied 1654 00000080:00200000:2.0:1775764923.302200:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000994cece5[3 0000000003208bfe 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.302203:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009f7f4b70 17ffffc0001001 3:0 ffff9d81c62fbb60 0 lru 00000080:00200000:2.0:1775764923.302210:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ff24bfe7 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 00000000a04df837 > 3< 0 0 > 4< 0 0 8 3899392 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.302212:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000994cece5 00000080:00200000:2.0:1775764923.302213:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.302217:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 1, from 0, to 1654 prio 0 00000008:00010000:2.0:1775764923.302230:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 000000004ec60dbe ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.302248:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f6578, count 1, dirtied 1 00000080:00200000:2.0:1775764923.302253:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 1 pages 1654 bytes, tot: 1654 00000080:00200000:2.0:1775764923.302255:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) run_tar.sh: write bytes 0, result: 1654 00000020:00010000:2.0:1775764923.302263:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.302265:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.302267:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420ab0 (ffff9d81912551b8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.302270:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.302279:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.302285:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.302286:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420ab0 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.302290:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 1654 bytes 00000080:00200000:2.0:1775764923.302292:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1775764923.302301:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x37:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.302303:0:34105:0:(file.c:2206:ll_file_io_generic()) run_tar.sh: 2 io complete with rc: 0, result: 1654, restart: 0 00000080:00200000:2.0:1775764923.302307:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x37:0x0] updating i_size 1654 i_blocks 0 00000080:00200000:2.0:1775764923.302309:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 1654 00000080:00200008:2.0:1775764923.302324:0:34105:0:(file.c:447:ll_file_release()) START file run_tar.sh:[0x240004281:0x37:0x0](ffff9d81843f63c8), flags 100001 00000002:00100000:2.0:1775764923.302342:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81874eb800 x1862023066197248/t38654705867(38654705867) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.302348:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.302351:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81874e9f80 x1862023066197888/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.302360:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874e9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066197888:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.302389:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9900 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.302614:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066197632:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.303004:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874e9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066197888:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.303018:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81874eb800 x1862023066197248/t38654705867(38654705867) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.303050:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.303058:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.303064:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.303070:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.303074:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.303077:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.303079:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.303081:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.303083:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.303087:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.303092:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.303095:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.303097:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.303099:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.303100:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.303102:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=run_tar.sh, flags=8 00000080:00200008:2.0:1775764923.303105:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file run_tar.sh:[0x240004281:0x37:0x0](ffff9d81843f63c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.303107:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x37:0x0](ffff9d81843f63c8),name=run_tar.sh 00000002:00010000:2.0:1775764923.303112:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x37:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.303115:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.303120:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.303124:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.303126:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x37:0x0](ffff9d81843f63c8) 00000080:00010000:2.0:1775764923.303127:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x37:0x0](ffff9d81843f63c8) for lock 0xeef44110eff54b7 00000080:00010000:2.0:1775764923.303129:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54b7 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.303131:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.303135:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.303148:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x37:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.303151:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x37:0x0] 00000008:00010000:2.0:1775764923.303159:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.303160:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420ab0, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.303166:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.303182:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.303187:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.303189:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.303194:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000fc7e3ef/0xeef44110eff54c5 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x21c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321554f4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.303203:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.303206:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x37:0x0] updating i_size 1654 i_blocks 0 00000020:00010000:2.0:1775764923.303208:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.303210:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.303212:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420ab0 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.303213:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.303215:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420ab0 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.303224:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x37:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.303249:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.303254:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.303258:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.303263:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.303267:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.303269:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.303271:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.303273:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.303275:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.303279:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.303287:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.303289:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.303291:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.303292:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.303294:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.303300:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-benchmark.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.303302:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.303307:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-benchmark.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.303320:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.303324:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.303329:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.303336:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874e8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066198016:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.303363:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.304090:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874e8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066198016:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.304097:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304105:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304111:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304117:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304123:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304129:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.304132:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304138:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304143:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304146:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.304148:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304152:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff54cc lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.304166:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81874e8000 x1862023066198016/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.304184:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.304193:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.304210:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.304215:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.304219:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.304225:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.304229:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.304231:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.304232:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.304234:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.304236:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.304240:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.304245:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.304248:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.304250:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.304252:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.304253:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.304257:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-benchmark.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.304260:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.304266:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-benchmark.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.304283:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304289:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.304300:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.304311:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874e8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066198144:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.304346:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.305127:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874e8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066198144:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.305136:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305141:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305144:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305148:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305151:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305154:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d9400 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.305156:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305159:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305162:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305166:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.305168:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305184:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff54d3 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.305192:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81874e8000 x1862023066198144/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.305198:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.305205:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.305235:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.305240:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.305244:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.305249:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.305253:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.305255:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.305257:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.305259:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.305261:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.305265:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.305271:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.305273:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.305275:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.305277:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.305278:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.305283:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.305286:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-benchmark.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da900, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:2.0:1775764923.305289:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.305297:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-benchmark.sh,[0x240004281:0x38:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.305313:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.305317:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.305322:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.305325:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81874e8000 x1862023066198400/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.305330:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.305337:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874e8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066198400:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.305393:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.306467:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874e8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066198400:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.306478:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306493:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 4/0,1 mode: --/PR res: [0x240004281:0x38:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306499:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 4/0,1 mode: --/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306506:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306513:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306521:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306527:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306532:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.306539:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81874e8000 x1862023066198400/t38654705869(38654705869) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.306548:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81874e8000 x1862023066198400/t38654705869(38654705869) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.306557:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.306565:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306572:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/56 (0 0) 00000080:00200000:2.0:1775764923.306576:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.306581:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-benchmark.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.306597:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x38:0x0](ffff9d81843f3248) 00000080:00200000:2.0:1775764923.306650:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x38:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.306657:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x38:0x0](ffff9d81843f3248) 00000080:00010000:2.0:1775764923.306659:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x38:0x0](ffff9d81843f3248) for lock 0xeef44110eff54da 00000080:00010000:2.0:1775764923.306662:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x38:0x0](ffff9d81843f3248) 00000080:00200008:2.0:1775764923.306667:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-benchmark.sh:[0x240004281:0x38:0x0](ffff9d81843f3248), flags 100301 00000002:00100000:2.0:1775764923.306676:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81874e8000 x1862023066198400/t38654705869(38654705869) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.306695:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54da from it ffff9d8190e6e420 00010000:00010000:2.0:1775764923.306697:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306703:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.306710:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x38:0x0](ffff9d81843f3248), inode mode 81a4 mask 40 00000080:00200008:2.0:1775764923.306717:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-benchmark.sh:[0x240004281:0x38:0x0](ffff9d81843f3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.306719:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x38:0x0](ffff9d81843f3248),name=sanity-benchmark.sh 00000002:00010000:2.0:1775764923.306726:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x38:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.306731:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.306736:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.306739:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.306741:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x38:0x0](ffff9d81843f3248) 00000080:00010000:2.0:1775764923.306743:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x38:0x0](ffff9d81843f3248) for lock 0xeef44110eff54da 00000080:00010000:2.0:1775764923.306744:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54da from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.306746:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.306750:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.306762:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x38:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.306765:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x38:0x0] 00000008:00010000:2.0:1775764923.306775:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.306776:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420430, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.306791:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.306797:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.306817:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.306847:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066198528:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.307527:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.307537:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.307544:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.307550:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.307557:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.307562:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.307570:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.307575:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.307583:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.307588:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.307591:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x38:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.307594:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000100:00100000:1.0:1775764923.307598:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066198528:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.307599:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.307603:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420430 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.307608:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.307610:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420430 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.307623:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x38:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.307645:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-benchmark.sh:[0x240004281:0x38:0x0], ppos: 0, count: 6548 00000080:00200000:2.0:1775764923.307655:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-benchmark.sh: write ppos: 0, bytes: 6548 00000080:00200000:2.0:1775764923.307657:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-benchmark.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.307661:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@000000007fab3529[0x110, 1, [0x240004281:0x38:0x0] hash] 00000020:00200000:2.0:1775764923.307662:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 6548) 0 0 00000080:00200000:2.0:1775764923.307664:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x38:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.307668:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 1] 00020000:00200000:2.0:1775764923.307671:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6548) 6548, 6548 00020000:00200000:2.0:1775764923.307673:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.307682:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 1] 00000008:00010000:2.0:1775764923.307686:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.307688:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420430, osc lock ffff9d8198381db0, flags 0x0 00010000:00010000:2.0:1775764923.307692:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/541 (0 8191) 00010000:00010000:2.0:1775764923.307704:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.307709:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.307714:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.307722:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874e9500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066198656:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.307746:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9e40 going to sleep for 11 seconds 00010000:00010000:2.0:1775764923.308493:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.308499:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.308505:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8c00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.308509:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.308535:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.308539:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.308565:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.308572:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.308592:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.308599:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b88bff63/0xeef44110eff54e1 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155548 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.308613:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.308616:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.308619:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.308969:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874e9500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066198656:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.308978:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.308987:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.308991:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.309427:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.309437:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.309447:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.309456:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.309466:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.309473:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.309489:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-benchmark.sh: write [0, 6548) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.309498:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.309518:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.309524:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007797eaa6[3 00000000f48549f5 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.309526:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bcb2d56a 17ffffc0001001 3:0 ffff9d81c62fb9a0 0 lru 00000080:00200000:2.0:1775764923.309532:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004bdf5a18 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 00000000950d7bf8 > 3< 0 0 > 4< 0 0 8 2797568 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.309533:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007797eaa6 00000080:00200000:2.0:1775764923.309534:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.309535:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 2452 bytes 00000080:00200000:2.0:1775764923.309549:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 2452, copied 2452 00000080:00200000:2.0:1775764923.309552:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b650827f[3 00000000f48549f5 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.309553:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000181e931c 17ffffc0001001 3:0 ffff9d81c62fb540 1 lru 00000080:00200000:2.0:1775764923.309557:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002423edec 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 00000000950d7bf8 > 3< 0 0 > 4< 0 0 8 2797568 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.309558:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b650827f 00000080:00200000:2.0:1775764923.309559:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.309561:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 2, from 0, to 2452 prio 0 00000008:00010000:2.0:1775764923.309569:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000ad8517d2 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.309578:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ad8517d2 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.309585:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f33f8, count 2, dirtied 2 00000080:00200000:2.0:1775764923.309587:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 2 pages 6548 bytes, tot: 6548 00000080:00200000:2.0:1775764923.309589:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-benchmark.sh: write bytes 0, result: 6548 00000020:00010000:2.0:1775764923.309594:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.309596:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.309598:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420430 (ffff9d81912551b8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.309600:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.309609:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.309614:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.309616:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420430 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.309619:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 6548 bytes 00000080:00200000:2.0:1775764923.309621:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:2.0:1775764923.309630:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x38:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.309632:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-benchmark.sh: 2 io complete with rc: 0, result: 6548, restart: 0 00000080:00200000:2.0:1775764923.309635:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x38:0x0] updating i_size 6548 i_blocks 0 00000080:00200000:2.0:1775764923.309637:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 6548 00000080:00200008:2.0:1775764923.309651:0:34105:0:(file.c:447:ll_file_release()) START file sanity-benchmark.sh:[0x240004281:0x38:0x0](ffff9d81843f3248), flags 100001 00000002:00100000:2.0:1775764923.309669:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81874e8000 x1862023066198400/t38654705869(38654705869) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.309676:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.309679:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81874e9500 x1862023066199040/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.309687:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874e9500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199040:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.309715:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9e40 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.310387:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874e9500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199040:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.310398:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81874e8000 x1862023066198400/t38654705869(38654705869) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.310422:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.310428:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.310434:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.310440:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.310444:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.310447:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.310448:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.310451:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.310452:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.310456:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.310461:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.310464:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.310466:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.310468:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.310469:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.310471:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-benchmark.sh, flags=8 00000080:00200008:2.0:1775764923.310474:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-benchmark.sh:[0x240004281:0x38:0x0](ffff9d81843f3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.310476:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x38:0x0](ffff9d81843f3248),name=sanity-benchmark.sh 00000002:00010000:2.0:1775764923.310481:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x38:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.310484:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.310492:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.310495:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.310497:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x38:0x0](ffff9d81843f3248) 00000080:00010000:2.0:1775764923.310498:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x38:0x0](ffff9d81843f3248) for lock 0xeef44110eff54da 00000080:00010000:2.0:1775764923.310500:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54da from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.310501:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.310505:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.310519:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x38:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.310522:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x38:0x0] 00000008:00010000:2.0:1775764923.310530:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.310532:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420430, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.310537:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.310543:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.310548:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.310550:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.310557:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000018769814/0xeef44110eff54e8 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215554f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.310562:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.310565:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x38:0x0] updating i_size 6548 i_blocks 0 00000020:00010000:2.0:1775764923.310567:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.310568:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.310569:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420430 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.310571:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.310573:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420430 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.310581:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x38:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.310604:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.310609:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.310613:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.310618:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.310621:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.310623:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.310625:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.310627:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.310629:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.310636:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.310641:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.310642:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.310644:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.310646:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.310647:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.310653:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-compr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.310655:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-compr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.310660:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-compr.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.310673:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.310677:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.310681:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.310688:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874ebb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199168:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.310715:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9e40 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.311410:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874ebb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199168:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.311416:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311421:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311425:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311429:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311432:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311435:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8200 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.311437:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311440:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311443:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311447:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.311449:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311452:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff54ef lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.311459:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81874ebb80 x1862023066199168/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.311465:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.311472:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.311488:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.311492:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.311496:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.311501:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.311505:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.311507:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.311509:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.311511:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.311513:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.311517:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.311521:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.311523:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.311525:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.311527:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.311528:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.311532:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-compr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.311533:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-compr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.311537:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-compr.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.311548:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311555:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.311559:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.311566:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874ebb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199296:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.311590:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9e40 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.312420:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874ebb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199296:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.312426:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312431:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312434:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312437:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312441:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312444:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d9e00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.312445:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312448:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312452:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312455:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.312458:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312461:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9e00/0xeef44110eff54f6 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.312467:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81874ebb80 x1862023066199296/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.312472:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.312479:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.312516:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.312524:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.312530:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.312539:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.312545:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.312549:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.312551:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.312555:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.312558:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.312570:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.312579:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.312583:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.312587:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.312589:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.312592:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.312600:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.312604:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-compr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da180, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:2.0:1775764923.312610:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-compr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.312621:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-compr.sh,[0x240004281:0x39:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.312643:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.312648:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.312652:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.312656:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81874ebb80 x1862023066199552/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.312660:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.312668:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874ebb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199552:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.312720:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9e40 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.313756:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874ebb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199552:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.313764:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313773:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 4/0,1 mode: --/PR res: [0x240004281:0x39:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313777:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 4/0,1 mode: --/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313782:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313787:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313791:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313795:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313800:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.313805:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81874ebb80 x1862023066199552/t38654705871(38654705871) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.313811:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81874ebb80 x1862023066199552/t38654705871(38654705871) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.313817:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.313823:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313827:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/57 (0 0) 00000080:00200000:2.0:1775764923.313830:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.313834:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-compr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.313844:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x39:0x0](ffff9d81843f42c8) 00000080:00200000:2.0:1775764923.313925:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x39:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.313935:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x39:0x0](ffff9d81843f42c8) 00000080:00010000:2.0:1775764923.313938:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x39:0x0](ffff9d81843f42c8) for lock 0xeef44110eff54fd 00000080:00010000:2.0:1775764923.313944:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x39:0x0](ffff9d81843f42c8) 00000080:00200008:2.0:1775764923.313950:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-compr.sh:[0x240004281:0x39:0x0](ffff9d81843f42c8), flags 100301 00000002:00100000:2.0:1775764923.313964:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81874ebb80 x1862023066199552/t38654705871(38654705871) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.313988:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54fd from it ffff9d8190e6e420 00010000:00010000:2.0:1775764923.313991:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.313996:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.314004:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x39:0x0](ffff9d81843f42c8), inode mode 81a4 mask 40 00000080:00200008:2.0:1775764923.314011:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-compr.sh:[0x240004281:0x39:0x0](ffff9d81843f42c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.314014:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x39:0x0](ffff9d81843f42c8),name=sanity-compr.sh 00000002:00010000:2.0:1775764923.314020:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x39:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.314025:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.314030:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.314034:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.314036:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x39:0x0](ffff9d81843f42c8) 00000080:00010000:2.0:1775764923.314037:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x39:0x0](ffff9d81843f42c8) for lock 0xeef44110eff54fd 00000080:00010000:2.0:1775764923.314039:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54fd from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.314041:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.314045:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.314057:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x39:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.314060:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x39:0x0] 00000008:00010000:2.0:1775764923.314069:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.314070:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420730, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.314085:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff5504 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.314090:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff5504 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.314117:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1775764923.314839:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x39:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.314843:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.314846:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.314848:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420730 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.314851:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.314853:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420730 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.314865:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x39:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.314898:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-compr.sh:[0x240004281:0x39:0x0], ppos: 0, count: 4750 00000080:00200000:2.0:1775764923.314908:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-compr.sh: write ppos: 0, bytes: 4750 00000080:00200000:2.0:1775764923.314911:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-compr.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.314914:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000308c912b[0x110, 1, [0x240004281:0x39:0x0] hash] 00000020:00200000:2.0:1775764923.314915:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 4750) 0 0 00000080:00200000:2.0:1775764923.314917:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x39:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.314921:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 1] 00020000:00200000:2.0:1775764923.314924:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 4750) 4750, 4750 00020000:00200000:2.0:1775764923.314926:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.314935:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 1] 00000008:00010000:2.0:1775764923.314940:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.314943:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420730, osc lock ffff9d8198381db0, flags 0x0 00010000:00010000:2.0:1775764923.314949:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/541 (0 8191) 00010000:00010000:2.0:1775764923.314966:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.314974:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.314981:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.314995:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199808:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.315023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9a80 going to sleep for 11 seconds 00010000:00010000:2.0:1775764923.315724:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.315730:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff5504 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321555a3 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.315739:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8600 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.315743:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff5504 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321555a3 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.315769:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.315773:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.315799:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff5504 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321555a3 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.315803:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.315817:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff5504 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321555a3 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.315822:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005fba3633/0xeef44110eff5504 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321555a3 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.315830:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.315831:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.315833:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.316213:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066199808:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.316224:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.316233:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.316239:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.316640:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.316658:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.316667:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.316674:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.316680:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.316684:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.316698:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-compr.sh: write [0, 4750) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.316705:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.316733:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.316738:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000084624d6a[3 0000000099ffc1bf 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.316740:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002cd18ee8 17ffffc0001021 2:0 ffff9d81c62fb000 0 lru 00000080:00200000:2.0:1775764923.316745:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004031c127 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 000000000f0ea87d > 3< 0 0 > 4< 0 0 8 3870720 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.316746:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000084624d6a 00000080:00200000:2.0:1775764923.316747:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.316748:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 654 bytes 00000080:00200000:2.0:1775764923.316761:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 654, copied 654 00000080:00200000:2.0:1775764923.316764:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dd1747cb[3 0000000099ffc1bf 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.316766:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c43db061 17ffffc0001001 3:0 ffff9d81c62fb2a0 1 lru 00000080:00200000:2.0:1775764923.316770:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e5620a7b 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 000000000f0ea87d > 3< 0 0 > 4< 0 0 8 3870720 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.316772:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dd1747cb 00000080:00200000:2.0:1775764923.316772:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.316774:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 2, from 0, to 654 prio 0 00000008:00010000:2.0:1775764923.316781:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 000000000302527d ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.316789:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000302527d ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.316795:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f4478, count 2, dirtied 2 00000080:00200000:2.0:1775764923.316798:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 2 pages 4750 bytes, tot: 4750 00000080:00200000:2.0:1775764923.316800:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-compr.sh: write bytes 0, result: 4750 00000020:00010000:2.0:1775764923.316805:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.316807:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.316809:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420730 (ffff9d81912551b8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.316811:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.316817:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.316822:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.316824:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420730 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.316827:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 4750 bytes 00000080:00200000:2.0:1775764923.316830:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:2.0:1775764923.316838:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x39:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.316840:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-compr.sh: 2 io complete with rc: 0, result: 4750, restart: 0 00000080:00200000:2.0:1775764923.316844:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x39:0x0] updating i_size 4750 i_blocks 0 00000080:00200000:2.0:1775764923.316846:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 4750 00000080:00200008:2.0:1775764923.316863:0:34105:0:(file.c:447:ll_file_release()) START file sanity-compr.sh:[0x240004281:0x39:0x0](ffff9d81843f42c8), flags 100001 00000002:00100000:2.0:1775764923.316906:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81874ebb80 x1862023066199552/t38654705871(38654705871) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.316917:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.316921:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81874eaa00 x1862023066200192/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.316935:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200192:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.316977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9a80 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.317157:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066199936:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.317582:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eaa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200192:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.317593:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81874ebb80 x1862023066199552/t38654705871(38654705871) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.317615:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.317621:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.317626:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.317632:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.317636:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.317639:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.317641:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.317645:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.317647:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.317654:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.317662:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.317666:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.317670:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.317673:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.317676:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.317679:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-compr.sh, flags=8 00000080:00200008:2.0:1775764923.317683:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-compr.sh:[0x240004281:0x39:0x0](ffff9d81843f42c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.317687:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x39:0x0](ffff9d81843f42c8),name=sanity-compr.sh 00000002:00010000:2.0:1775764923.317695:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x39:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.317702:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.317715:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.317721:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.317725:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x39:0x0](ffff9d81843f42c8) 00000080:00010000:2.0:1775764923.317728:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x39:0x0](ffff9d81843f42c8) for lock 0xeef44110eff54fd 00000080:00010000:2.0:1775764923.317731:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff54fd from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.317734:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.317741:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.317758:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x39:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.317762:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x39:0x0] 00000008:00010000:2.0:1775764923.317770:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.317771:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420730, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.317777:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.317783:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.317788:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.317790:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.317798:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000bcb15a9e/0xeef44110eff550b lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x21d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321555aa expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.317803:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.317806:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x39:0x0] updating i_size 4750 i_blocks 0 00000020:00010000:2.0:1775764923.317807:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.317809:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.317810:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420730 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.317812:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.317813:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420730 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.317822:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x39:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.317845:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.317850:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.317854:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.317859:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.317863:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.317866:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.317867:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.317869:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.317871:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.317905:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.317911:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.317913:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.317916:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.317917:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.317919:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.317924:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-dom.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.317927:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-dom.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.317931:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-dom.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.317944:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.317948:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.317953:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.317960:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200320:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.317995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9a80 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.318727:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200320:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.318733:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318738:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318742:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318746:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318749:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318752:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d9400 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.318754:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318757:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318760:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318764:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.318766:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318769:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5512 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.318775:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81874eb100 x1862023066200320/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.318781:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.318788:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.318804:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.318809:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.318812:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.318817:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.318822:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.318825:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.318827:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.318830:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.318833:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.318840:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.318849:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.318853:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.318856:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.318859:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.318861:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.318867:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-dom.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.318871:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-dom.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.318898:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-dom.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.318913:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318920:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.318924:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.318931:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200448:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.318960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9a80 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.319735:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200448:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.319741:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319746:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319749:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319752:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319756:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319759:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.319761:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319764:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319767:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319771:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.319773:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319776:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff5519 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.319782:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81874eb100 x1862023066200448/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.319787:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.319794:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.319821:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.319825:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.319829:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.319834:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.319838:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.319841:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.319842:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.319844:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.319846:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.319854:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.319859:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.319861:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.319863:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.319864:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.319865:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.319870:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.319872:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-dom.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da000, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:2.0:1775764923.319892:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-dom.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.319900:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-dom.sh,[0x240004281:0x3a:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.319919:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.319925:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.319932:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.319937:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81874eb100 x1862023066200704/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.319945:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.319957:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200704:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.320022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9a80 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.320981:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81874eb100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200704:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.320988:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.320997:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321001:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321006:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321011:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321016:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321020:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321024:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.321029:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81874eb100 x1862023066200704/t38654705873(38654705873) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.321035:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81874eb100 x1862023066200704/t38654705873(38654705873) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.321041:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.321056:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321063:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/58 (0 0) 00000080:00200000:2.0:1775764923.321067:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.321072:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-dom.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.321086:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x3a:0x0](ffff9d81843f3a88) 00000080:00200000:2.0:1775764923.321140:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x3a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.321147:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x3a:0x0](ffff9d81843f3a88) 00000080:00010000:2.0:1775764923.321148:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3a:0x0](ffff9d81843f3a88) for lock 0xeef44110eff5520 00000080:00010000:2.0:1775764923.321152:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3a:0x0](ffff9d81843f3a88) 00000080:00200008:2.0:1775764923.321156:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-dom.sh:[0x240004281:0x3a:0x0](ffff9d81843f3a88), flags 100301 00000002:00100000:2.0:1775764923.321165:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81874eb100 x1862023066200704/t38654705873(38654705873) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.321194:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5520 from it ffff9d8190e6e420 00010000:00010000:2.0:1775764923.321197:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321202:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.321210:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x3a:0x0](ffff9d81843f3a88), inode mode 81a4 mask 40 00000080:00200008:2.0:1775764923.321216:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-dom.sh:[0x240004281:0x3a:0x0](ffff9d81843f3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.321220:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3a:0x0](ffff9d81843f3a88),name=sanity-dom.sh 00000002:00010000:2.0:1775764923.321226:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3a:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.321231:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.321236:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.321239:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.321241:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3a:0x0](ffff9d81843f3a88) 00000080:00010000:2.0:1775764923.321243:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3a:0x0](ffff9d81843f3a88) for lock 0xeef44110eff5520 00000080:00010000:2.0:1775764923.321245:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5520 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.321247:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.321251:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.321263:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.321266:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3a:0x0] 00000008:00010000:2.0:1775764923.321275:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.321276:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420e30, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.321301:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.321307:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.321329:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.321361:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066200832:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.321982:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.321991:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.321999:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.322005:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.322012:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.322019:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.322027:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.322032:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.322037:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.322042:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.322047:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3a:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.322050:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.322055:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.322057:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420e30 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000100:00100000:1.0:1775764923.322059:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066200832:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.322061:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.322065:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420e30 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.322079:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.322099:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-dom.sh:[0x240004281:0x3a:0x0], ppos: 0, count: 6821 00000080:00200000:2.0:1775764923.322108:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-dom.sh: write ppos: 0, bytes: 6821 00000080:00200000:2.0:1775764923.322111:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-dom.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.322113:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@000000002a761e20[0x110, 1, [0x240004281:0x3a:0x0] hash] 00000020:00200000:2.0:1775764923.322115:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 6821) 0 0 00000080:00200000:2.0:1775764923.322116:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3a:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.322120:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 1] 00020000:00200000:2.0:1775764923.322123:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 6821) 6821, 6821 00020000:00200000:2.0:1775764923.322125:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.322134:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 1] 00000008:00010000:2.0:1775764923.322138:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.322140:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420e30, osc lock ffff9d8198381db0, flags 0x0 00010000:00010000:2.0:1775764923.322144:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/542 (0 8191) 00010000:00010000:2.0:1775764923.322156:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.322161:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.322166:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.322187:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200960:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.322210:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.322896:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.322905:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082326272 00000100:00100000:1.0:1775764923.322908:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082326272 00000100:00100000:1.0:1775764923.322930:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 269 00000100:00100000:1.0:1775764923.322935:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819049f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082326272:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.322941:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.322972:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.322977:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819049f050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082326272:12345-192.168.202.105@tcp:104: Request processed in 42us (167us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.322981:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 269 00010000:00010000:2.0:1775764923.323003:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.323013:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.323019:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8a00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.323022:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.323051:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.323055:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.323089:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.323095:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.323109:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.323113:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5527 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321555fe expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.323124:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.323127:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.323130:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.323131:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066201088:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.323519:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066200960:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.323527:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.323533:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.323537:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.323914:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.323926:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.323933:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.323940:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.323947:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.323954:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.323972:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-dom.sh: write [0, 6821) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.323980:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.324002:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.324008:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fc3fd377[3 000000008b1bc456 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.324010:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000087cecc54 17ffffc0001001 3:0 ffff9d81c62fbee0 0 lru 00000080:00200000:2.0:1775764923.324015:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000083fa4d75 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 00000000fbb55f58 > 3< 0 0 > 4< 0 0 8 2764800 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.324016:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fc3fd377 00000080:00200000:2.0:1775764923.324017:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.324019:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 2725 bytes 00000080:00200000:2.0:1775764923.324044:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 2725, copied 2725 00000080:00200000:2.0:1775764923.324047:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a0750ea2[3 000000008b1bc456 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.324050:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000eb8dfb9 17ffffc0001001 3:0 ffff9d81c62fbd20 1 lru 00000080:00200000:2.0:1775764923.324053:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002d9866c6 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 00000000fbb55f58 > 3< 0 0 > 4< 0 0 8 2764800 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.324055:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a0750ea2 00000080:00200000:2.0:1775764923.324055:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.324058:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 2, from 0, to 2725 prio 0 00000008:00010000:2.0:1775764923.324065:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 000000003c673f1d ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.324074:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000003c673f1d ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.324081:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f3c38, count 2, dirtied 2 00000080:00200000:2.0:1775764923.324083:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 2 pages 6821 bytes, tot: 6821 00000080:00200000:2.0:1775764923.324085:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-dom.sh: write bytes 0, result: 6821 00000020:00010000:2.0:1775764923.324091:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.324092:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.324094:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420e30 (ffff9d81912551b8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.324097:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.324102:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.324108:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.324109:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420e30 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.324113:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 6821 bytes 00000080:00200000:2.0:1775764923.324116:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:2.0:1775764923.324124:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3a:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.324126:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-dom.sh: 2 io complete with rc: 0, result: 6821, restart: 0 00000080:00200000:2.0:1775764923.324130:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3a:0x0] updating i_size 6821 i_blocks 0 00000080:00200000:2.0:1775764923.324132:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 6821 00000080:00200008:2.0:1775764923.324145:0:34105:0:(file.c:447:ll_file_release()) START file sanity-dom.sh:[0x240004281:0x3a:0x0](ffff9d81843f3a88), flags 100001 00000002:00100000:2.0:1775764923.324162:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81874eb100 x1862023066200704/t38654705873(38654705873) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.324168:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.324186:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fa680 x1862023066201344/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.324200:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fa680 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066201344:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.324233:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.324520:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906faa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066201088:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.324814:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fa680 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066201344:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.324825:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81874eb100 x1862023066200704/t38654705873(38654705873) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.324849:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.324854:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.324860:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.324867:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.324871:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.324892:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.324895:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.324897:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.324900:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.324905:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.324911:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.324914:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.324917:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.324918:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.324920:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.324921:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-dom.sh, flags=8 00000080:00200008:2.0:1775764923.324925:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-dom.sh:[0x240004281:0x3a:0x0](ffff9d81843f3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.324927:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3a:0x0](ffff9d81843f3a88),name=sanity-dom.sh 00000002:00010000:2.0:1775764923.324932:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3a:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.324936:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.324944:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.324948:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.324951:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3a:0x0](ffff9d81843f3a88) 00000080:00010000:2.0:1775764923.324953:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3a:0x0](ffff9d81843f3a88) for lock 0xeef44110eff5520 00000080:00010000:2.0:1775764923.324956:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5520 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.324959:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.324965:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.324986:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.324990:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3a:0x0] 00000008:00010000:2.0:1775764923.325003:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.325005:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d8187420e30, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.325012:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.325019:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.325024:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.325026:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.325038:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000fed64898/0xeef44110eff552e lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155605 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.325046:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.325050:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3a:0x0] updating i_size 6821 i_blocks 0 00000020:00010000:2.0:1775764923.325054:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.325056:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.325058:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8187420e30 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.325061:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.325063:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8187420e30 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.325076:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.325107:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.325113:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.325118:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.325123:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.325127:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.325129:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.325131:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.325133:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.325135:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.325143:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.325148:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.325150:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.325151:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.325153:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.325154:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.325159:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-flr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.325161:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-flr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.325165:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-flr.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.325193:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.325197:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.325202:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.325209:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066201472:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.325236:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd96c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.325976:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066201472:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.325985:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.325993:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.325999:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326004:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326011:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326016:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8c00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.326019:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326025:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326031:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326037:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.326041:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326046:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff5535 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.326058:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f9f80 x1862023066201472/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.326067:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.326077:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.326098:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.326104:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.326108:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.326113:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.326117:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.326120:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.326121:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.326123:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.326125:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.326129:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.326134:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.326136:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.326139:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.326140:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.326141:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.326145:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-flr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.326147:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-flr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.326151:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-flr.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.326162:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326169:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.326186:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.326194:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066201600:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.326221:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd96c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.327058:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066201600:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.327067:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327074:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327080:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327086:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327092:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327096:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d9400 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.327099:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327105:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327111:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327116:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.327119:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327124:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff553c lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.327131:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f9f80 x1862023066201600/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.327136:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.327143:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.327169:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.327188:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.327192:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.327197:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.327201:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.327204:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.327205:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.327208:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.327209:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.327217:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.327222:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.327225:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.327227:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.327228:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.327230:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.327234:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.327237:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-flr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da180, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.327240:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-flr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.327248:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-flr.sh,[0x240004281:0x3b:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.327262:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.327266:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.327271:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.327274:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f9f80 x1862023066201856/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.327279:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.327285:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066201856:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.327336:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd96c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.328374:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066201856:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.328385:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328396:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328401:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328406:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328411:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328417:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328428:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328432:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.328438:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906f9f80 x1862023066201856/t38654705875(38654705875) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.328444:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f9f80 x1862023066201856/t38654705875(38654705875) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.328451:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.328457:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328462:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/59 (0 0) 00000080:00200000:2.0:1775764923.328464:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.328468:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-flr.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.328480:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x3b:0x0](ffff9d81843f0908) 00000080:00200000:2.0:1775764923.328528:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x3b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.328533:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x3b:0x0](ffff9d81843f0908) 00000080:00010000:2.0:1775764923.328535:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3b:0x0](ffff9d81843f0908) for lock 0xeef44110eff5543 00000080:00010000:2.0:1775764923.328539:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3b:0x0](ffff9d81843f0908) 00000080:00200008:2.0:1775764923.328543:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-flr.sh:[0x240004281:0x3b:0x0](ffff9d81843f0908), flags 100301 00000002:00100000:2.0:1775764923.328551:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f9f80 x1862023066201856/t38654705875(38654705875) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.328568:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5543 from it ffff9d8190e6e420 00010000:00010000:2.0:1775764923.328571:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328576:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.328584:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x3b:0x0](ffff9d81843f0908), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.328591:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-flr.sh:[0x240004281:0x3b:0x0](ffff9d81843f0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.328593:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3b:0x0](ffff9d81843f0908),name=sanity-flr.sh 00000002:00010000:2.0:1775764923.328599:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3b:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.328604:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.328609:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.328613:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.328615:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3b:0x0](ffff9d81843f0908) 00000080:00010000:2.0:1775764923.328616:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3b:0x0](ffff9d81843f0908) for lock 0xeef44110eff5543 00000080:00010000:2.0:1775764923.328618:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5543 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.328620:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.328627:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.328645:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.328650:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3b:0x0] 00000008:00010000:2.0:1775764923.328663:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.328666:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81874205b0, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.328685:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b981df3b/0xeef44110eff554a lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.328692:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b981df3b/0xeef44110eff554a lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.328711:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1775764923.329473:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3b:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.329478:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.329482:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.329484:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81874205b0 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.329486:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.329491:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81874205b0 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.329507:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.329595:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-flr.sh:[0x240004281:0x3b:0x0], ppos: 0, count: 141987 00000080:00200000:2.0:1775764923.329607:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-flr.sh: write ppos: 0, bytes: 141987 00000080:00200000:2.0:1775764923.329610:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-flr.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.329613:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@000000001f674a3e[0x110, 1, [0x240004281:0x3b:0x0] hash] 00000020:00200000:2.0:1775764923.329614:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 141987) 0 0 00000080:00200000:2.0:1775764923.329616:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3b:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.329620:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 34] 00020000:00200000:2.0:1775764923.329623:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 141987) 141987, 141987 00020000:00200000:2.0:1775764923.329627:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.329639:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 34] 00000008:00010000:2.0:1775764923.329645:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.329647:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81874205b0, osc lock ffff9d8198381db0, flags 0x0 00010000:00010000:2.0:1775764923.329653:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/542 (0 143359) 00010000:00010000:2.0:1775764923.329670:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.329679:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->143359] (req 0->143359) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.329685:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->143359] (req 0->143359) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.329695:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066202112:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.329719:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd90c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.330376:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082326528 00000100:00100000:1.0:1775764923.330381:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082326528 00000100:00100000:1.0:1775764923.330411:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 271 00000100:00100000:1.0:1775764923.330420:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082326528:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.330429:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b981df3b/0xeef44110eff554a lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155659 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.330463:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.330467:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082326528:12345-192.168.202.105@tcp:104: Request processed in 50us (134us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.330472:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 271 00010000:00010000:2.0:1775764923.330484:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.330490:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b981df3b/0xeef44110eff554a lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155659 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.330496:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8200 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.330499:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b981df3b/0xeef44110eff554a lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155659 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.330522:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.330528:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.330565:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b981df3b/0xeef44110eff554a lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155659 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.330573:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.330591:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b981df3b/0xeef44110eff554a lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155659 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.330599:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b981df3b/0xeef44110eff554a lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155659 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.330610:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00010000:00010000:2.0:1775764923.330612:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.330615:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.330618:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.330619:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066202240:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.331087:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066202112:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.331096:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->143359] (req 0->143359) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.331102:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->143359] (req 0->143359) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.331107:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->143359] (req 0->143359) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.331486:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.331499:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.331508:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.331515:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.331521:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.331526:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.331538:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-flr.sh: write [0, 141987) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.331544:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331567:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331572:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004e60f4cb[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331574:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000df284db5 17ffffc0001001 3:0 ffff9d818935d540 0 lru 00000080:00200000:2.0:1775764923.331578:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000057f3082c 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331580:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004e60f4cb 00000080:00200000:2.0:1775764923.331580:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.331582:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331593:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331595:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007c5e0d90[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331597:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001e4ed47a 17ffffc0001001 3:0 ffff9d818935d000 1 lru 00000080:00200000:2.0:1775764923.331601:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a0e638a2 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331602:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007c5e0d90 00000080:00200000:2.0:1775764923.331603:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.331604:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331611:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331613:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a609908e[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331614:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bde539f3 17ffffc0001001 3:0 ffff9d818935d2a0 2 lru 00000080:00200000:2.0:1775764923.331618:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008be51d6c 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331619:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a609908e 00000080:00200000:2.0:1775764923.331619:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.331620:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331627:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331630:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ecf12383[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331631:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000073bf56fe 17ffffc0001001 3:0 ffff9d818935dee0 3 lru 00000080:00200000:2.0:1775764923.331635:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000132ecedf 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331636:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ecf12383 00000080:00200000:2.0:1775764923.331637:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.331638:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331645:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331647:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c73e32a2[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331649:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004ae2be84 17ffffc0001001 3:0 ffff9d818935dd20 4 lru 00000080:00200000:2.0:1775764923.331652:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a58eb5d9 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331653:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c73e32a2 00000080:00200000:2.0:1775764923.331653:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.331654:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331661:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331663:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000069ba647a[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331665:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000015dbc9c9 17ffffc0001001 3:0 ffff9d818935d7e0 5 lru 00000080:00200000:2.0:1775764923.331668:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dc9a3d57 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331669:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000069ba647a 00000080:00200000:2.0:1775764923.331669:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.331670:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331676:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331678:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000074651327[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331680:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000715073d8 17ffffc0001001 3:0 ffff9d818935d8c0 6 lru 00000080:00200000:2.0:1775764923.331683:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000070db2917 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331684:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000074651327 00000080:00200000:2.0:1775764923.331685:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.331686:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331692:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331694:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000038984087[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331695:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000017992a1b 17ffffc0001001 3:0 ffff9d818935dc40 7 lru 00000080:00200000:2.0:1775764923.331698:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000927d9904 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331699:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000038984087 00000080:00200000:2.0:1775764923.331700:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.331700:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331707:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331709:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003018da97[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331716:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b243db5d 17ffffc0001001 3:0 ffff9d818935da80 8 lru 00000080:00200000:2.0:1775764923.331720:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d093c45e 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331721:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003018da97 00000080:00200000:2.0:1775764923.331722:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.331723:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331731:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331733:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000de2d489c[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331735:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002120b30c 17ffffc0001001 3:0 ffff9d818935d380 9 lru 00000080:00200000:2.0:1775764923.331738:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009a0a1729 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331740:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000de2d489c 00000080:00200000:2.0:1775764923.331741:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.331743:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331759:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331763:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000833f5c9a[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331766:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000025c4d19b 17ffffc0001021 2:0 ffff9d818935de00 10 lru 00000080:00200000:2.0:1775764923.331771:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004f4efa61 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331773:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000833f5c9a 00000080:00200000:2.0:1775764923.331774:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.331776:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331789:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331793:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a67bfa45[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331795:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000045b6db29 17ffffc0001001 3:0 ffff9d818935d1c0 11 lru 00000080:00200000:2.0:1775764923.331800:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008d9cb9bf 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331802:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a67bfa45 00000080:00200000:2.0:1775764923.331803:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.331804:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331822:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331826:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ef4f5213[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331828:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f116de48 17ffffc0001001 3:0 ffff9d818935d700 12 lru 00000080:00200000:2.0:1775764923.331834:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000c5fde48 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331835:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ef4f5213 00000080:00200000:2.0:1775764923.331836:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.331838:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331853:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331857:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000729c2779[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331859:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000022629938 17ffffc0001001 3:0 ffff9d818935d620 13 lru 00000080:00200000:2.0:1775764923.331863:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000028b314b4 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331865:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000729c2779 00000080:00200000:2.0:1775764923.331866:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.331868:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331899:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331902:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000640aed2a[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331904:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008030f92e 17ffffc0001001 3:0 ffff9d818935d460 14 lru 00000080:00200000:2.0:1775764923.331909:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b36540f6 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331910:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000640aed2a 00000080:00200000:2.0:1775764923.331911:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.331913:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331924:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331926:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006cef7266[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331929:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b97af661 17ffffc0001001 3:0 ffff9d818935d0e0 15 lru 00000080:00200000:2.0:1775764923.331934:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dbf08896 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331936:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006cef7266 00000080:00200000:2.0:1775764923.331937:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.331938:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331949:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331953:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000389d5083[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331956:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e3cb922a 17ffffc0001001 3:0 ffff9d818935db60 16 lru 00000080:00200000:2.0:1775764923.331962:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f5043cfb 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331964:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000389d5083 00000080:00200000:2.0:1775764923.331965:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.331967:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.331979:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.331982:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005b901126[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.331984:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cb6c6b46 17ffffc0001001 3:0 ffff9d818935d9a0 17 lru 00000080:00200000:2.0:1775764923.331988:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007ff487a2 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.331989:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005b901126 00000080:00200000:2.0:1775764923.331990:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.331991:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332005:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332008:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000095fea3f9[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332010:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002854d007 17ffffc0001001 3:0 ffff9d818753a460 18 lru 00000080:00200000:2.0:1775764923.332013:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f3b013c1 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332015:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000095fea3f9 00000080:00200000:2.0:1775764923.332015:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.332017:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332026:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332028:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000026483659[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332030:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000061e331a7 17ffffc0001001 3:0 ffff9d818753a0e0 19 lru 00000080:00200000:2.0:1775764923.332034:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000069294a44 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332035:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000026483659 00000080:00200000:2.0:1775764923.332036:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.332037:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000100:00100000:1.0:1775764923.332042:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066202240:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.332052:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332055:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ec9edfec[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332058:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000003ede008 17ffffc0001001 3:0 ffff9d818753ab60 20 lru 00000080:00200000:2.0:1775764923.332061:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009c826767 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332062:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ec9edfec 00000080:00200000:2.0:1775764923.332062:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.332063:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332073:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332077:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005863718a[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332079:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cb004095 17ffffc0001001 3:0 ffff9d818753a9a0 21 lru 00000080:00200000:2.0:1775764923.332084:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f0b89b66 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332086:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005863718a 00000080:00200000:2.0:1775764923.332087:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.332089:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332102:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332106:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f78b4bfd[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332109:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a6479228 17ffffc0001001 3:0 ffff9d818753a540 22 lru 00000080:00200000:2.0:1775764923.332114:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000005523979 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332116:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f78b4bfd 00000080:00200000:2.0:1775764923.332117:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.332119:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332132:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332136:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a45ad541[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332139:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001059007a 17ffffc0001001 3:0 ffff9d818753a000 23 lru 00000080:00200000:2.0:1775764923.332144:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009da988ac 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332146:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a45ad541 00000080:00200000:2.0:1775764923.332147:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.332149:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332167:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332181:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009ae7d586[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332184:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003da46b90 17ffffc0001001 3:0 ffff9d818753a2a0 24 lru 00000080:00200000:2.0:1775764923.332189:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001f198a63 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332190:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009ae7d586 00000080:00200000:2.0:1775764923.332191:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.332192:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332205:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332207:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000007c51026[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332208:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007540f05d 17ffffc0001021 2:0 ffff9d818753aee0 25 lru 00000080:00200000:2.0:1775764923.332211:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ec1c3991 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332212:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000007c51026 00000080:00200000:2.0:1775764923.332213:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.332214:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332222:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332224:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000940a6984[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332225:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008ddfac86 17ffffc0001001 3:0 ffff9d818753ad20 26 lru 00000080:00200000:2.0:1775764923.332228:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004f7d196a 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332230:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000940a6984 00000080:00200000:2.0:1775764923.332231:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.332232:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332238:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332240:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000e6b2e71[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332242:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006debd527 17ffffc0001001 3:0 ffff9d818753a7e0 27 lru 00000080:00200000:2.0:1775764923.332245:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c016f5c3 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332246:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000e6b2e71 00000080:00200000:2.0:1775764923.332247:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.332248:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332254:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332256:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009eaa3292[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332264:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000c8c4b6b 17ffffc0001001 3:0 ffff9d818753a8c0 28 lru 00000080:00200000:2.0:1775764923.332269:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003c80cd69 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332270:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009eaa3292 00000080:00200000:2.0:1775764923.332271:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.332272:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332279:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332281:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f8e9e428[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332283:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006865c559 17ffffc0001001 3:0 ffff9d818753ac40 29 lru 00000080:00200000:2.0:1775764923.332286:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000050589108 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332288:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f8e9e428 00000080:00200000:2.0:1775764923.332288:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.332289:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332296:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332297:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f0a94943[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332299:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007e37309b 17ffffc0001001 3:0 ffff9d818753aa80 30 lru 00000080:00200000:2.0:1775764923.332302:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008f4e15b2 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332303:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f0a94943 00000080:00200000:2.0:1775764923.332304:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.332305:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332312:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332319:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c18a0205[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332320:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000062521406 17ffffc0001001 3:0 ffff9d818753a380 31 lru 00000080:00200000:2.0:1775764923.332324:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f1c8aa16 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332325:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c18a0205 00000080:00200000:2.0:1775764923.332326:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.332327:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332334:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332336:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000022a10214[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332337:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004c65c13e 17ffffc0001001 3:0 ffff9d818753ae00 32 lru 00000080:00200000:2.0:1775764923.332340:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000012b914c6 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332341:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000022a10214 00000080:00200000:2.0:1775764923.332342:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.332343:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.332351:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.332353:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008d80abae[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332354:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000405afafb 17ffffc0001001 3:0 ffff9d818753a1c0 33 lru 00000080:00200000:2.0:1775764923.332357:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007b57988c 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332359:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008d80abae 00000080:00200000:2.0:1775764923.332359:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.332360:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 2723 bytes 00000080:00200000:2.0:1775764923.332369:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 2723, copied 2723 00000080:00200000:2.0:1775764923.332371:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000074f89f27[3 00000000b080e7c4 2 1 00000000a58da78d] 00000080:00200000:2.0:1775764923.332372:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000092acb6b0 17ffffc0001001 3:0 ffff9d818753a700 34 lru 00000080:00200000:2.0:1775764923.332375:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006f4f73b9 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 00000000409b8422 0000000008d2d1a4 > 3< 0 0 > 4< 0 0 8 3837952 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.332376:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000074f89f27 00000080:00200000:2.0:1775764923.332377:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.332379:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 35, from 0, to 2723 prio 0 00000008:00010000:2.0:1775764923.332387:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332400:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332405:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332410:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332414:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332418:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332422:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332426:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332430:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332434:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332439:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332444:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332448:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332452:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332456:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.332463:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f0ab8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.332466:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332470:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332475:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332488:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332496:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332504:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332512:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332520:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332529:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332537:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332545:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332551:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332556:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332560:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332564:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.332571:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f0ab8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.332574:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332578:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332582:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332587:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.332592:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000038608d6 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.332600:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f0ab8, count 5, dirtied 5 00000080:00200000:2.0:1775764923.332602:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 35 pages 141987 bytes, tot: 141987 00000080:00200000:2.0:1775764923.332604:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-flr.sh: write bytes 0, result: 141987 00000020:00010000:2.0:1775764923.332610:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.332612:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.332614:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81874205b0 (ffff9d81912551b8/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.332616:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.332624:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.332633:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252a88 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.332636:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81874205b0 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.332641:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 141987 bytes 00000080:00200000:2.0:1775764923.332644:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 34] 00000080:00200000:2.0:1775764923.332656:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3b:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.332660:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-flr.sh: 2 io complete with rc: 0, result: 141987, restart: 0 00000080:00200000:2.0:1775764923.332665:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3b:0x0] updating i_size 141987 i_blocks 0 00000080:00200000:2.0:1775764923.332668:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 141987 00000080:00200008:2.0:1775764923.332687:0:34105:0:(file.c:447:ll_file_release()) START file sanity-flr.sh:[0x240004281:0x3b:0x0](ffff9d81843f0908), flags 100001 00000002:00100000:2.0:1775764923.332710:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906f9f80 x1862023066201856/t38654705875(38654705875) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.332719:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.332724:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f9500 x1862023066202496/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.332742:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066202496:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.332779:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd90c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.333431:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066202496:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.333445:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906f9f80 x1862023066201856/t38654705875(38654705875) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.333473:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.333481:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.333488:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.333496:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.333500:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.333503:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.333505:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.333507:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.333509:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.333514:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.333519:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.333521:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.333524:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.333526:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.333528:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.333530:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-flr.sh, flags=8 00000080:00200008:2.0:1775764923.333533:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-flr.sh:[0x240004281:0x3b:0x0](ffff9d81843f0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.333535:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3b:0x0](ffff9d81843f0908),name=sanity-flr.sh 00000002:00010000:2.0:1775764923.333540:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3b:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.333544:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.333549:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.333553:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.333555:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3b:0x0](ffff9d81843f0908) 00000080:00010000:2.0:1775764923.333556:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3b:0x0](ffff9d81843f0908) for lock 0xeef44110eff5543 00000080:00010000:2.0:1775764923.333558:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5543 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.333559:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.333563:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.333578:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.333581:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3b:0x0] 00000008:00010000:2.0:1775764923.333590:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.333591:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81874205b0, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.333596:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.333606:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.333611:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.333612:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.333616:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000076bb827b/0xeef44110eff5551 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x21e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->143359) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155660 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.333621:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.333624:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3b:0x0] updating i_size 141987 i_blocks 0 00000020:00010000:2.0:1775764923.333626:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.333627:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.333629:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81874205b0 (ffff9d81912551b8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.333631:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8188736fe8 (ffff9d81912553f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.333632:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81874205b0 (ffff9d81912553f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.333640:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.333663:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.333668:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.333672:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.333676:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.333684:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.333686:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.333687:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.333690:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.333691:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.333695:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.333699:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.333701:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.333703:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.333705:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.333706:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.333711:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-hsm.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.333713:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.333717:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-hsm.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.333730:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.333734:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.333738:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.333745:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066202624:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.333775:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd90c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.334617:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066202624:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.334625:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334633:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334639:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334645:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334651:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334656:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8188fb8e00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.334659:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334664:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334670:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334677:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.334680:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334689:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8e00/0xeef44110eff5558 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.334699:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fb800 x1862023066202624/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.334707:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.334716:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.334738:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.334746:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.334752:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.334759:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.334765:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.334769:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.334771:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.334775:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.334778:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.334785:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.334793:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.334797:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.334801:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.334804:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.334806:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.334812:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-hsm.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.334814:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.334819:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-hsm.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.334832:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334836:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.334840:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.334847:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066202752:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.334902:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd90c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.335644:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066202752:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.335650:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335655:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335658:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335662:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335668:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335671:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.335673:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335676:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335679:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335682:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.335684:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335688:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8800/0xeef44110eff555f lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.335694:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fb800 x1862023066202752/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.335699:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.335705:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.335730:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.335735:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.335738:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.335747:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.335751:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.335753:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.335755:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.335757:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.335759:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.335763:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.335767:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.335770:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.335772:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.335773:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.335775:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.335780:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.335782:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-hsm.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da900, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.335786:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.335792:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-hsm.sh,[0x240004281:0x3c:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.335806:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.335810:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.335815:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.335818:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fb800 x1862023066203008/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.335823:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.335830:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066203008:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.335896:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd90c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.336927:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fb800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066203008:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.336934:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.336942:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.336946:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.336951:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.336956:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.336961:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.336969:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.336973:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.336991:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fb800 x1862023066203008/t38654705877(38654705877) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.337000:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fb800 x1862023066203008/t38654705877(38654705877) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.337010:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.337019:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.337024:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/60 (0 0) 00000080:00200000:2.0:1775764923.337026:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.337030:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-hsm.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.337039:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x3c:0x0](ffff9d81843f4b08) 00000080:00200000:2.0:1775764923.337086:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x3c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.337092:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x3c:0x0](ffff9d81843f4b08) 00000080:00010000:2.0:1775764923.337094:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3c:0x0](ffff9d81843f4b08) for lock 0xeef44110eff5566 00000080:00010000:2.0:1775764923.337097:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3c:0x0](ffff9d81843f4b08) 00000080:00200008:2.0:1775764923.337102:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-hsm.sh:[0x240004281:0x3c:0x0](ffff9d81843f4b08), flags 100301 00000002:00100000:2.0:1775764923.337110:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fb800 x1862023066203008/t38654705877(38654705877) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.337126:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5566 from it ffff9d8190e6e420 00010000:00010000:2.0:1775764923.337128:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.337133:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.337141:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x3c:0x0](ffff9d81843f4b08), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.337148:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-hsm.sh:[0x240004281:0x3c:0x0](ffff9d81843f4b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.337150:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3c:0x0](ffff9d81843f4b08),name=sanity-hsm.sh 00000002:00010000:2.0:1775764923.337156:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3c:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.337160:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.337165:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.337168:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.337184:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3c:0x0](ffff9d81843f4b08) 00000080:00010000:2.0:1775764923.337186:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3c:0x0](ffff9d81843f4b08) for lock 0xeef44110eff5566 00000080:00010000:2.0:1775764923.337188:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5566 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.337190:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.337194:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.337208:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.337211:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3c:0x0] 00000008:00010000:2.0:1775764923.337220:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.337221:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce230, osc lock ffff9d8198381db0, flags 0x400001002 00010000:00010000:2.0:1775764923.337236:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.337242:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.337257:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.337280:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066203136:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.337954:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.337967:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.337978:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.337987:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.337997:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.338012:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1775764923.338024:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.338032:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.338040:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.338048:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1775764923.338059:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066203136:192.168.202.105@tcp:101:cp.0 00000100:00100000:1.0:1775764923.338943:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082326784 00000100:00100000:1.0:1775764923.338947:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082326784 00000100:00100000:1.0:1775764923.338969:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 273 00000100:00100000:1.0:1775764923.338974:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190709c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082326784:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.338983:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.339015:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.339019:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190709c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082326784:12345-192.168.202.105@tcp:104: Request processed in 46us (117us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.339024:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 273 00010000:00010000:2.0:1775764923.339043:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.339048:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.339057:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81902b9a00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.339061:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.339092:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.339098:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.339137:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 2/0,0 mode: --/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.339146:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.339183:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.339189:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066203392:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.339198:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff556d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321556b4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.339212:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.339214:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.339216:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.340017:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.340029:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.340037:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.340044:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.340050:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.340055:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.340067:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-hsm.sh: write [0, 168862) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.340074:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340100:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340107:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b90b7ff6[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340109:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fdca3f62 17ffffc0001001 3:0 ffff9d818753a620 0 lru 00000080:00200000:2.0:1775764923.340117:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000357d4b70 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340119:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b90b7ff6 00000080:00200000:2.0:1775764923.340120:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.340123:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340154:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340159:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ea25854d[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340162:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b72e81f7 17ffffc0001001 3:0 ffff9d8190737e00 1 lru 00000080:00200000:2.0:1775764923.340167:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000009027769 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340180:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ea25854d 00000080:00200000:2.0:1775764923.340182:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.340184:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340198:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340201:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005fb25b15[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340202:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b0f0f8e6 17ffffc0001001 3:0 ffff9d81907371c0 2 lru 00000080:00200000:2.0:1775764923.340206:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000017edb62b 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340207:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005fb25b15 00000080:00200000:2.0:1775764923.340208:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.340209:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340216:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340218:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000729f2f4b[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340220:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003944b45c 17ffffc0001001 3:0 ffff9d8190737700 3 lru 00000080:00200000:2.0:1775764923.340223:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002a8a35a8 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340225:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000729f2f4b 00000080:00200000:2.0:1775764923.340225:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.340226:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340233:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340235:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000067289388[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340237:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d8ebf1a4 17ffffc0001001 3:0 ffff9d8190737620 4 lru 00000080:00200000:2.0:1775764923.340240:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000373fef91 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340241:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000067289388 00000080:00200000:2.0:1775764923.340242:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.340243:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340254:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340265:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ed6d4c2b[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340267:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000391f0fa5 17ffffc0001021 2:0 ffff9d8190737460 5 lru 00000080:00200000:2.0:1775764923.340271:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d731f030 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340272:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ed6d4c2b 00000080:00200000:2.0:1775764923.340273:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.340274:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340282:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340284:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bcb033cf[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340286:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009574ffaa 17ffffc0001001 3:0 ffff9d81907370e0 6 lru 00000080:00200000:2.0:1775764923.340289:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a235f38f 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340290:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bcb033cf 00000080:00200000:2.0:1775764923.340291:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.340292:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340298:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340300:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bbeb0b2f[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340302:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e2f46a7e 17ffffc0001001 3:0 ffff9d8190737b60 7 lru 00000080:00200000:2.0:1775764923.340305:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e5b1b575 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340306:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bbeb0b2f 00000080:00200000:2.0:1775764923.340307:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.340308:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340315:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340317:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b80b57ef[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340318:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009f014409 17ffffc0001001 3:0 ffff9d81907379a0 8 lru 00000080:00200000:2.0:1775764923.340324:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000091672527 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340326:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b80b57ef 00000080:00200000:2.0:1775764923.340327:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.340328:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340334:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340336:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000234fc21d[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340338:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006a70b796 17ffffc0001001 3:0 ffff9d8190737540 9 lru 00000080:00200000:2.0:1775764923.340341:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fdc567b4 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340342:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000234fc21d 00000080:00200000:2.0:1775764923.340343:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.340344:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340351:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340352:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000454d0909[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340354:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003fe65a8e 17ffffc0001001 3:0 ffff9d8190737000 10 lru 00000080:00200000:2.0:1775764923.340357:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007bdfad07 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340358:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000454d0909 00000080:00200000:2.0:1775764923.340358:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.340359:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340366:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340368:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a02a6575[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340369:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000085062bd5 17ffffc0001001 3:0 ffff9d81907372a0 11 lru 00000080:00200000:2.0:1775764923.340372:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000086d9627a 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340373:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a02a6575 00000080:00200000:2.0:1775764923.340374:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.340375:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340382:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340384:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000049f11fe6[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340385:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d862b061 17ffffc0001001 3:0 ffff9d8190737ee0 12 lru 00000080:00200000:2.0:1775764923.340389:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004bb4192e 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340390:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000049f11fe6 00000080:00200000:2.0:1775764923.340391:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.340391:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340398:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340400:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005d16ea3f[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340402:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ab3419a8 17ffffc0001001 3:0 ffff9d8190737d20 13 lru 00000080:00200000:2.0:1775764923.340405:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a12bedf2 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340406:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005d16ea3f 00000080:00200000:2.0:1775764923.340406:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.340407:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340414:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340416:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b8c3918d[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340417:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003a3c4efb 17ffffc0001001 3:0 ffff9d81907377e0 14 lru 00000080:00200000:2.0:1775764923.340420:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007596487f 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340422:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b8c3918d 00000080:00200000:2.0:1775764923.340422:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.340423:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340430:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340433:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000050c3e647[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340435:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003e08f5a0 17ffffc0001001 3:0 ffff9d81907378c0 15 lru 00000080:00200000:2.0:1775764923.340438:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ea87449b 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340440:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000050c3e647 00000080:00200000:2.0:1775764923.340440:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.340441:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340449:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340451:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002803cab5[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340453:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005d11ee38 17ffffc0001001 3:0 ffff9d8190737c40 16 lru 00000080:00200000:2.0:1775764923.340456:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000068e2fbf1 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340457:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002803cab5 00000080:00200000:2.0:1775764923.340458:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.340459:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340467:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340470:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c8765e64[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340471:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000092d8118a 17ffffc0001001 3:0 ffff9d8190737a80 17 lru 00000080:00200000:2.0:1775764923.340475:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c8eef73e 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340476:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c8765e64 00000080:00200000:2.0:1775764923.340477:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.340478:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340485:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340487:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000093c31d9d[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340489:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009bb2f841 17ffffc0001001 3:0 ffff9d8190737380 18 lru 00000080:00200000:2.0:1775764923.340492:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000062e6390c 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340494:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000093c31d9d 00000080:00200000:2.0:1775764923.340495:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.340496:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340510:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340512:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000028e78ac5[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340515:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003cdc31b6 17ffffc0001001 3:0 ffff9d8186ee9ee0 19 lru 00000100:00100000:1.0:1775764923.340516:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066203392:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.340524:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000078649736 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340526:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000028e78ac5 00000080:00200000:2.0:1775764923.340527:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.340528:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340538:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340540:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b7a1941c[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340541:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b6119b20 17ffffc0001021 2:0 ffff9d8186ee9d20 20 lru 00000080:00200000:2.0:1775764923.340544:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d09c1a44 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340546:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b7a1941c 00000080:00200000:2.0:1775764923.340546:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.340547:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340557:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340560:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000ce43883[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340563:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003999f7f0 17ffffc0001001 3:0 ffff9d8186ee97e0 21 lru 00000080:00200000:2.0:1775764923.340568:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000f1ccfda 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340570:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000ce43883 00000080:00200000:2.0:1775764923.340571:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.340573:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340586:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340590:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c18137d2[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340592:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000e071d91 17ffffc0001001 3:0 ffff9d8186ee98c0 22 lru 00000080:00200000:2.0:1775764923.340597:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006cdaada7 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340599:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c18137d2 00000080:00200000:2.0:1775764923.340600:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.340602:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340615:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340617:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000075927146[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340619:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f1433377 17ffffc0001001 3:0 ffff9d8186ee9c40 23 lru 00000080:00200000:2.0:1775764923.340623:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000021ba6628 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340625:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000075927146 00000080:00200000:2.0:1775764923.340626:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.340627:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340639:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340642:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000090febe34[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340645:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f8773da0 17ffffc0001001 3:0 ffff9d8186ee9a80 24 lru 00000080:00200000:2.0:1775764923.340650:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007a9963a1 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340652:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000090febe34 00000080:00200000:2.0:1775764923.340653:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.340655:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340666:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340669:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000081d68233[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340672:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ed195b4a 17ffffc0001001 3:0 ffff9d8186ee9380 25 lru 00000080:00200000:2.0:1775764923.340677:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007db95944 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340679:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000081d68233 00000080:00200000:2.0:1775764923.340679:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.340681:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340690:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340693:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007ea5dde3[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340694:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f8cabdfb 17ffffc0001001 3:0 ffff9d8186ee9e00 26 lru 00000080:00200000:2.0:1775764923.340697:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005bbc85cb 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340699:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007ea5dde3 00000080:00200000:2.0:1775764923.340699:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.340700:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340707:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340709:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000068a3b7b5[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340710:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e975a1d4 17ffffc0001001 3:0 ffff9d8186ee91c0 27 lru 00000080:00200000:2.0:1775764923.340713:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000059018342 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340715:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000068a3b7b5 00000080:00200000:2.0:1775764923.340715:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.340716:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340722:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340724:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000097778c5[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340726:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002eaad3d6 17ffffc0001001 3:0 ffff9d8186ee9700 28 lru 00000080:00200000:2.0:1775764923.340729:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bbe6daf1 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340730:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000097778c5 00000080:00200000:2.0:1775764923.340731:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.340732:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340738:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340740:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bd92d6a8[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340742:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002c832804 17ffffc0001001 3:0 ffff9d8186ee9620 29 lru 00000080:00200000:2.0:1775764923.340745:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e28a86b0 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340746:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bd92d6a8 00000080:00200000:2.0:1775764923.340746:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.340747:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340753:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340755:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000061f14fbe[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340757:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000033dd9724 17ffffc0001001 3:0 ffff9d8186ee9460 30 lru 00000080:00200000:2.0:1775764923.340760:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004c8bf18a 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340761:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000061f14fbe 00000080:00200000:2.0:1775764923.340762:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.340763:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340769:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340771:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a0f35b8d[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340772:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008eb17d74 17ffffc0001001 3:0 ffff9d8186ee90e0 31 lru 00000080:00200000:2.0:1775764923.340775:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ccfc3836 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340776:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a0f35b8d 00000080:00200000:2.0:1775764923.340777:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.340778:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340784:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340786:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ca002447[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340788:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000094492027 17ffffc0001001 3:0 ffff9d8186ee9b60 32 lru 00000080:00200000:2.0:1775764923.340792:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fe5157e0 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340794:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ca002447 00000080:00200000:2.0:1775764923.340795:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.340796:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340810:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340814:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004eb5393f[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340816:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cd799e87 17ffffc0001001 3:0 ffff9d8186ee99a0 33 lru 00000080:00200000:2.0:1775764923.340822:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d5d77754 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340823:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004eb5393f 00000080:00200000:2.0:1775764923.340824:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.340826:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340837:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340839:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001525fbb8[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340841:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b80403c6 17ffffc0001001 3:0 ffff9d8186ee9540 34 lru 00000080:00200000:2.0:1775764923.340844:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000eb3c0c10 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340845:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001525fbb8 00000080:00200000:2.0:1775764923.340846:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.340847:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340856:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340858:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006725feb9[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340860:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000098789944 17ffffc0001021 2:0 ffff9d8186ee9000 35 lru 00000080:00200000:2.0:1775764923.340863:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000076ae852d 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340865:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006725feb9 00000080:00200000:2.0:1775764923.340865:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.340866:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340873:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340890:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000027541821[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340893:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003675b062 17ffffc0001001 3:0 ffff9d8186ee92a0 36 lru 00000080:00200000:2.0:1775764923.340898:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d7eacf09 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340900:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000027541821 00000080:00200000:2.0:1775764923.340901:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.340903:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340922:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340926:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cf4d740a[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340928:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cef4063c 17ffffc0001001 3:0 ffff9d8190667d20 37 lru 00000080:00200000:2.0:1775764923.340934:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000032f7734f 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340936:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cf4d740a 00000080:00200000:2.0:1775764923.340936:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.340938:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340953:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340957:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008da85f88[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340959:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000417305c2 17ffffc0001001 3:0 ffff9d81906677e0 38 lru 00000080:00200000:2.0:1775764923.340965:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000707e8da6 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340967:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008da85f88 00000080:00200000:2.0:1775764923.340968:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.340970:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340981:0:34105:0:(rw26.c:858:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.340983:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002ef7ea6c[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.340985:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000041fe5ee3 17ffffc0001001 3:0 ffff9d81906678c0 39 lru 00000080:00200000:2.0:1775764923.340988:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000041991d7f 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.340990:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002ef7ea6c 00000080:00200000:2.0:1775764923.340990:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1775764923.340991:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.340999:0:34105:0:(rw26.c:858:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.341001:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c90201af[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.341002:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000c5c709f 17ffffc0001001 3:0 ffff9d8190667c40 40 lru 00000080:00200000:2.0:1775764923.341005:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000082d53eeb 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.341007:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c90201af 00000080:00200000:2.0:1775764923.341007:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1775764923.341008:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 41 of 0 to 926 bytes 00000080:00200000:2.0:1775764923.341017:0:34105:0:(rw26.c:858:ll_write_end()) pos 167936, len 926, copied 926 00000080:00200000:2.0:1775764923.341019:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d4cfe46b[3 00000000460fb0cd 2 1 000000009f833b1e] 00000080:00200000:2.0:1775764923.341020:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000903b674f 17ffffc0001001 3:0 ffff9d8190667a80 41 lru 00000080:00200000:2.0:1775764923.341023:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000565ddc97 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 000000009452db5f 00000000d601def0 > 3< 0 0 > 4< 0 0 8 2732032 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.341025:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d4cfe46b 00000080:00200000:2.0:1775764923.341026:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1775764923.341027:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 42, from 0, to 926 prio 0 00000008:00010000:2.0:1775764923.341036:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341045:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341050:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341055:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341063:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341067:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341071:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341075:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341079:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341083:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341087:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341091:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341095:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341100:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341104:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.341112:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f4cb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.341115:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341119:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341123:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341128:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341132:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341136:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341143:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341148:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341152:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341156:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341160:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341164:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341168:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341186:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341190:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.341197:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f4cb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.341200:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341205:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341209:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341213:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341217:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341221:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341225:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341229:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341236:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341241:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341245:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.341249:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c0fa1157 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.341254:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f4cb8, count 12, dirtied 12 00000080:00200000:2.0:1775764923.341257:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 42 pages 168862 bytes, tot: 168862 00000080:00200000:2.0:1775764923.341259:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-hsm.sh: write bytes 0, result: 168862 00000020:00010000:2.0:1775764923.341266:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8186f44998 (ffff9d81856e5488/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.341269:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8186f44998 (ffff9d81856e5488/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.341271:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81861995b0 (ffff9d81856e5248/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.341275:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.341282:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.341291:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8186f44998 (ffff9d81856e5488/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.341293:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81861995b0 (ffff9d81856e5488/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.341299:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 168862 bytes 00000080:00200000:2.0:1775764923.341303:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 41] 00000080:00200000:2.0:1775764923.341320:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3c:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.341322:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-hsm.sh: 2 io complete with rc: 0, result: 168862, restart: 0 00000080:00200000:2.0:1775764923.341326:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3c:0x0] updating i_size 168862 i_blocks 0 00000080:00200000:2.0:1775764923.341329:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 168862 00000080:00200008:2.0:1775764923.341343:0:34105:0:(file.c:447:ll_file_release()) START file sanity-hsm.sh:[0x240004281:0x3c:0x0](ffff9d81843f4b08), flags 100001 00000002:00100000:2.0:1775764923.341359:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fb800 x1862023066203008/t38654705877(38654705877) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.341366:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.341368:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fb100 x1862023066203648/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.341377:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fb100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066203648:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.341407:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9000 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.341992:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fb100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066203648:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.342003:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fb800 x1862023066203008/t38654705877(38654705877) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.342031:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.342039:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.342047:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.342055:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.342062:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.342067:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.342068:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.342071:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.342073:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.342078:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.342084:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.342087:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.342089:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.342092:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.342094:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.342097:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-hsm.sh, flags=8 00000080:00200008:2.0:1775764923.342102:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-hsm.sh:[0x240004281:0x3c:0x0](ffff9d81843f4b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.342106:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3c:0x0](ffff9d81843f4b08),name=sanity-hsm.sh 00000002:00010000:2.0:1775764923.342114:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3c:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.342121:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.342128:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.342135:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.342139:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3c:0x0](ffff9d81843f4b08) 00000080:00010000:2.0:1775764923.342141:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3c:0x0](ffff9d81843f4b08) for lock 0xeef44110eff5566 00000080:00010000:2.0:1775764923.342144:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5566 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.342147:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.342155:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.342194:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.342198:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3c:0x0] 00000008:00010000:2.0:1775764923.342207:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.342209:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ceb30, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.342215:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.342221:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.342227:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.342229:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.342233:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000011d36cc7/0xeef44110eff5574 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->172031) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321556bb expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.342238:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.342241:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3c:0x0] updating i_size 168862 i_blocks 0 00000020:00010000:2.0:1775764923.342243:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.342244:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.342246:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ceb30 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.342248:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.342249:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ceb30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.342261:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.342295:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.342304:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.342310:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.342318:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.342325:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.342328:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.342331:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.342335:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.342337:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.342344:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.342352:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.342354:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.342357:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.342358:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.342360:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.342365:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-krb5.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.342368:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.342373:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-krb5.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.342387:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.342391:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.342395:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.342403:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066203776:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.342438:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9000 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.343167:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066203776:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.343186:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343191:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343195:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343198:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343202:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343205:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8c00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.343207:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343211:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343215:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343218:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.343220:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343223:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff557b lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.343230:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f8e00 x1862023066203776/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.343236:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.343243:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.343258:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.343262:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.343266:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.343271:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.343275:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.343277:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.343278:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.343280:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.343282:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.343286:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.343291:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.343293:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.343295:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.343296:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.343298:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.343301:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-krb5.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.343303:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.343307:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-krb5.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.343318:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343322:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.343326:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.343333:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066203904:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.343365:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9000 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.344098:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066203904:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.344107:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344118:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344123:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344129:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344135:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344139:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8200 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.344141:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344144:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344148:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344151:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.344153:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344157:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff5582 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.344164:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f8e00 x1862023066203904/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.344186:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.344193:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.344221:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.344225:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.344230:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.344235:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.344239:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.344241:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.344242:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.344245:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.344246:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.344250:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.344255:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.344257:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.344259:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.344261:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.344263:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.344268:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.344271:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-krb5.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da000, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.344275:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.344282:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-krb5.sh,[0x240004281:0x3d:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.344300:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.344307:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.344313:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.344319:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f8e00 x1862023066204160/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.344326:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.344337:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066204160:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.344396:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9000 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.345385:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066204160:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.345393:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345401:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345406:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345411:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345419:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345424:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345428:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345432:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.345437:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906f8e00 x1862023066204160/t38654705879(38654705879) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.345442:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f8e00 x1862023066204160/t38654705879(38654705879) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.345448:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.345453:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345458:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/61 (0 0) 00000080:00200000:2.0:1775764923.345460:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.345464:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-krb5.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.345474:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x3d:0x0](ffff9d81843f21c8) 00000080:00200000:2.0:1775764923.345517:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x3d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.345523:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x3d:0x0](ffff9d81843f21c8) 00000080:00010000:2.0:1775764923.345524:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3d:0x0](ffff9d81843f21c8) for lock 0xeef44110eff5589 00000080:00010000:2.0:1775764923.345528:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3d:0x0](ffff9d81843f21c8) 00000080:00200008:2.0:1775764923.345533:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-krb5.sh:[0x240004281:0x3d:0x0](ffff9d81843f21c8), flags 100301 00000002:00100000:2.0:1775764923.345540:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f8e00 x1862023066204160/t38654705879(38654705879) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.345557:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5589 from it ffff9d8190e6e240 00010000:00010000:2.0:1775764923.345560:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345565:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.345572:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x3d:0x0](ffff9d81843f21c8), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.345578:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-krb5.sh:[0x240004281:0x3d:0x0](ffff9d81843f21c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.345581:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3d:0x0](ffff9d81843f21c8),name=sanity-krb5.sh 00000002:00010000:2.0:1775764923.345587:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3d:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.345591:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.345596:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.345600:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.345602:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3d:0x0](ffff9d81843f21c8) 00000080:00010000:2.0:1775764923.345603:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3d:0x0](ffff9d81843f21c8) for lock 0xeef44110eff5589 00000080:00010000:2.0:1775764923.345605:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5589 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.345607:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.345611:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.345623:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.345628:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3d:0x0] 00000008:00010000:2.0:1775764923.345641:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.345643:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce9b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.345664:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5590 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.345673:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5590 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.345695:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1775764923.346601:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.346605:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.346609:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.346611:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce9b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.346614:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.346616:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce9b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.346630:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.346662:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-krb5.sh:[0x240004281:0x3d:0x0], ppos: 0, count: 31711 00000080:00200000:2.0:1775764923.346677:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-krb5.sh: write ppos: 0, bytes: 31711 00000080:00200000:2.0:1775764923.346680:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-krb5.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.346684:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000f06c49ba[0x110, 1, [0x240004281:0x3d:0x0] hash] 00000020:00200000:2.0:1775764923.346686:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 31711) 0 0 00000080:00200000:2.0:1775764923.346687:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.346692:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 7] 00020000:00200000:2.0:1775764923.346694:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 31711) 31711, 31711 00020000:00200000:2.0:1775764923.346697:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.346706:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 7] 00000008:00010000:2.0:1775764923.346710:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.346712:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce9b0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.346716:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/543 (0 32767) 00010000:00010000:2.0:1775764923.346728:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.346734:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.346739:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.346747:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fad80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066204416:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.346770:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd99c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.347451:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.347459:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082327040 00000100:00100000:1.0:1775764923.347462:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082327040 00000100:00100000:1.0:1775764923.347484:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 275 00000100:00100000:1.0:1775764923.347490:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082327040:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.347499:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5590 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215570f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.347537:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.347548:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796d850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082327040:12345-192.168.202.105@tcp:104: Request processed in 53us (136us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.347554:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 275 00010000:00010000:2.0:1775764923.347561:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.347567:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5590 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215570f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.347576:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8a00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.347580:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5590 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa3215570f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.347611:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.347617:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.347651:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5590 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215570f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.347660:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.347679:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5590 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215570f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.347685:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff5590 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215570f expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.347694:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.347697:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.347700:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.348183:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fad80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066204416:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.348193:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.348200:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.348206:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->32767] (req 0->32767) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.348578:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.348588:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.348595:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.348608:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.348621:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.348626:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.348638:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-krb5.sh: write [0, 31711) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.348646:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.348670:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.348677:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003bbc08b3[3 00000000cd8a93e7 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.348680:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000df9f5140 17ffffc0001001 3:0 ffff9d8190667380 0 lru 00000080:00200000:2.0:1775764923.348686:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008e4e92ab 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 000000005c9c8d9d > 3< 0 0 > 4< 0 0 8 3670016 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.348688:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003bbc08b3 00000080:00200000:2.0:1775764923.348689:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.348692:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.348712:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.348716:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005248c7a6[3 00000000cd8a93e7 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.348719:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001a6dd939 17ffffc0001001 3:0 ffff9d8190667e00 1 lru 00000080:00200000:2.0:1775764923.348724:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000076ceb5be 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 000000005c9c8d9d > 3< 0 0 > 4< 0 0 8 3670016 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.348726:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005248c7a6 00000080:00200000:2.0:1775764923.348727:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.348729:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.348741:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.348743:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000aa2ce7c0[3 00000000cd8a93e7 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.348745:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b2bff1fb 17ffffc0001001 3:0 ffff9d81906671c0 2 lru 00000080:00200000:2.0:1775764923.348748:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000043fb3a22 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000409b8422 000000005c9c8d9d > 3< 0 0 > 4< 0 0 8 3670016 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.348749:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000aa2ce7c0 00000080:00200000:2.0:1775764923.348749:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.348750:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.348757:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.348759:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000086c36696[3 00000000cd8a93e7 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.348761:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c6b38464 17ffffc0001001 3:0 ffff9d8190667700 3 lru 00000080:00200000:2.0:1775764923.348764:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000069d09c3d 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000409b8422 000000005c9c8d9d > 3< 0 0 > 4< 0 0 8 3670016 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.348765:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000086c36696 00000080:00200000:2.0:1775764923.348766:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.348767:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.348773:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.348775:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b56ff205[3 00000000cd8a93e7 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.348777:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d1b54139 17ffffc0001001 3:0 ffff9d8190667620 4 lru 00000080:00200000:2.0:1775764923.348780:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000990980b0 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000409b8422 000000005c9c8d9d > 3< 0 0 > 4< 0 0 8 3670016 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.348781:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b56ff205 00000080:00200000:2.0:1775764923.348781:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.348782:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.348792:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.348796:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000035437ac1[3 00000000cd8a93e7 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.348798:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f11b25f7 17ffffc0001001 3:0 ffff9d8190667460 5 lru 00000080:00200000:2.0:1775764923.348803:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005050ea07 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000409b8422 000000005c9c8d9d > 3< 0 0 > 4< 0 0 8 3670016 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.348804:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000035437ac1 00000080:00200000:2.0:1775764923.348805:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.348808:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.348821:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.348825:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ecfc780d[3 00000000cd8a93e7 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.348827:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000025fb2166 17ffffc0001001 3:0 ffff9d81906670e0 6 lru 00000080:00200000:2.0:1775764923.348833:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000eda183ea 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000409b8422 000000005c9c8d9d > 3< 0 0 > 4< 0 0 8 3670016 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.348834:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ecfc780d 00000080:00200000:2.0:1775764923.348835:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.348837:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 3039 bytes 00000080:00200000:2.0:1775764923.348849:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 3039, copied 3039 00000080:00200000:2.0:1775764923.348852:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e5a54565[3 00000000cd8a93e7 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.348853:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003b676693 17ffffc0001001 3:0 ffff9d8190667b60 7 lru 00000080:00200000:2.0:1775764923.348856:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ffb33b8e 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000409b8422 000000005c9c8d9d > 3< 0 0 > 4< 0 0 8 3670016 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.348858:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e5a54565 00000080:00200000:2.0:1775764923.348858:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.348861:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 8, from 0, to 3039 prio 0 00000008:00010000:2.0:1775764923.348868:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 0000000019ed4e00 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.348892:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000019ed4e00 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.348898:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000019ed4e00 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.348902:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000019ed4e00 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.348907:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000019ed4e00 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.348914:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000019ed4e00 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.348918:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000019ed4e00 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.348922:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000019ed4e00 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.348929:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f2378, count 8, dirtied 8 00000080:00200000:2.0:1775764923.348931:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 8 pages 31711 bytes, tot: 31711 00000080:00200000:2.0:1775764923.348933:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-krb5.sh: write bytes 0, result: 31711 00000020:00010000:2.0:1775764923.348939:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.348941:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.348942:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce9b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.348945:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.348950:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.348959:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.348962:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce9b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.348967:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 31711 bytes 00000080:00200000:2.0:1775764923.348970:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 7] 00000080:00200000:2.0:1775764923.348983:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.348987:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-krb5.sh: 2 io complete with rc: 0, result: 31711, restart: 0 00000080:00200000:2.0:1775764923.348992:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3d:0x0] updating i_size 31711 i_blocks 0 00000080:00200000:2.0:1775764923.348995:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 31711 00000080:00200008:2.0:1775764923.349014:0:34105:0:(file.c:447:ll_file_release()) START file sanity-krb5.sh:[0x240004281:0x3d:0x0](ffff9d81843f21c8), flags 100001 00000002:00100000:2.0:1775764923.349032:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906f8e00 x1862023066204160/t38654705879(38654705879) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.349039:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.349043:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fad80 x1862023066204800/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.349053:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fad80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066204800:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.349089:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd99c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.349139:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066204544:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.349739:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fad80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066204800:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.349749:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906f8e00 x1862023066204160/t38654705879(38654705879) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.349770:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.349776:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.349782:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.349788:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.349795:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.349799:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.349802:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.349806:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.349809:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.349816:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.349825:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.349829:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.349833:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.349835:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.349837:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.349839:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-krb5.sh, flags=8 00000080:00200008:2.0:1775764923.349842:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-krb5.sh:[0x240004281:0x3d:0x0](ffff9d81843f21c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.349845:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3d:0x0](ffff9d81843f21c8),name=sanity-krb5.sh 00000002:00010000:2.0:1775764923.349851:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3d:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.349855:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.349860:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.349864:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.349866:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3d:0x0](ffff9d81843f21c8) 00000080:00010000:2.0:1775764923.349868:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3d:0x0](ffff9d81843f21c8) for lock 0xeef44110eff5589 00000080:00010000:2.0:1775764923.349870:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5589 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.349872:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.349894:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.349908:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.349912:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3d:0x0] 00000008:00010000:2.0:1775764923.349921:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.349922:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce9b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.349927:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.349933:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.349939:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.349940:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.349945:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004bc6897a/0xeef44110eff5597 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x21f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155716 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.349950:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.349954:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3d:0x0] updating i_size 31711 i_blocks 0 00000020:00010000:2.0:1775764923.349957:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.349959:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.349962:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce9b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.349964:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.349967:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce9b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.349979:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.350011:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.350017:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.350022:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.350027:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.350031:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.350033:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.350035:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.350037:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.350040:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.350046:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.350054:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.350057:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.350060:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.350063:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.350065:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.350073:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-lfsck.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.350076:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.350084:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lfsck.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.350101:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.350106:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.350110:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.350118:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066204928:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.350147:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd99c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.351037:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066204928:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.351046:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351054:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351059:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351065:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351071:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351077:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190f1a200 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.351080:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351089:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351093:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351097:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.351099:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351103:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff559e lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.351111:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f8700 x1862023066204928/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.351117:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.351124:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.351140:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.351144:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.351148:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.351153:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.351157:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.351159:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.351160:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.351162:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.351164:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.351168:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.351188:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.351190:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.351192:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.351194:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.351195:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.351200:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-lfsck.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.351202:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.351207:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lfsck.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.351218:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351222:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.351227:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.351233:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066205056:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.351258:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd99c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.352111:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066205056:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.352118:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352126:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352130:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352133:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352137:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352140:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190f1bc00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.352141:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352145:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352148:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352151:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.352153:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352157:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff55a5 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.352163:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f8700 x1862023066205056/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.352184:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.352192:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.352219:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.352224:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.352229:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.352234:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.352238:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.352241:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.352242:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.352245:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.352247:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.352251:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.352256:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.352259:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.352261:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.352262:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.352264:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.352268:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.352271:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-lfsck.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da180, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.352274:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.352282:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lfsck.sh,[0x240004281:0x3e:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.352298:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.352305:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.352312:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.352326:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f8700 x1862023066205312/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.352334:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.352346:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066205312:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.352414:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd99c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.353416:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066205312:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.353423:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353431:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353435:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353441:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353449:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353454:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353458:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353462:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.353467:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906f8700 x1862023066205312/t38654705881(38654705881) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.353472:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f8700 x1862023066205312/t38654705881(38654705881) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.353478:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.353483:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353487:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/62 (0 0) 00000080:00200000:2.0:1775764923.353490:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.353494:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-lfsck.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.353504:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x3e:0x0](ffff9d81843f00c8) 00000080:00200000:2.0:1775764923.353547:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x3e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.353552:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x3e:0x0](ffff9d81843f00c8) 00000080:00010000:2.0:1775764923.353554:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3e:0x0](ffff9d81843f00c8) for lock 0xeef44110eff55ac 00000080:00010000:2.0:1775764923.353557:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3e:0x0](ffff9d81843f00c8) 00000080:00200008:2.0:1775764923.353562:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-lfsck.sh:[0x240004281:0x3e:0x0](ffff9d81843f00c8), flags 100301 00000002:00100000:2.0:1775764923.353569:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f8700 x1862023066205312/t38654705881(38654705881) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.353587:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55ac from it ffff9d8190e6e240 00010000:00010000:2.0:1775764923.353590:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353595:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.353604:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x3e:0x0](ffff9d81843f00c8), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.353611:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-lfsck.sh:[0x240004281:0x3e:0x0](ffff9d81843f00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.353614:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3e:0x0](ffff9d81843f00c8),name=sanity-lfsck.sh 00000002:00010000:2.0:1775764923.353620:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3e:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.353624:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.353630:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.353633:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.353636:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3e:0x0](ffff9d81843f00c8) 00000080:00010000:2.0:1775764923.353637:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3e:0x0](ffff9d81843f00c8) for lock 0xeef44110eff55ac 00000080:00010000:2.0:1775764923.353640:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55ac from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.353642:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.353647:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.353658:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.353662:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3e:0x0] 00000008:00010000:2.0:1775764923.353671:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.353672:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ceab0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.353687:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.353692:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.353711:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.353732:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066205440:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.354399:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.354408:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.354414:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.354419:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.354425:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.354431:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.354439:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.354443:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.354448:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.354453:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.354459:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3e:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.354463:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000100:00100000:1.0:1775764923.354466:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fa300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066205440:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.354466:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.354471:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ceab0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.354474:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.354476:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ceab0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.354488:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.354564:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-lfsck.sh:[0x240004281:0x3e:0x0], ppos: 0, count: 215320 00000080:00200000:2.0:1775764923.354574:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-lfsck.sh: write ppos: 0, bytes: 215320 00000080:00200000:2.0:1775764923.354578:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-lfsck.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.354581:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000df941a6f[0x110, 1, [0x240004281:0x3e:0x0] hash] 00000020:00200000:2.0:1775764923.354582:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 215320) 0 0 00000080:00200000:2.0:1775764923.354584:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3e:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.354588:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 52] 00020000:00200000:2.0:1775764923.354591:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 215320) 215320, 215320 00020000:00200000:2.0:1775764923.354594:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.354602:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 52] 00000008:00010000:2.0:1775764923.354607:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.354608:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ceab0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.354613:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/544 (0 217087) 00010000:00010000:2.0:1775764923.354626:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 3/0,1 mode: --/PW res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.354631:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 3/0,1 mode: --/PW res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.354636:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 3/0,1 mode: --/PW res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.354644:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066205568:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.354672:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9b40 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.355295:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082327296 00000100:00100000:1.0:1775764923.355299:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082327296 00000100:00100000:1.0:1775764923.355326:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 277 00000100:00100000:1.0:1775764923.355331:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082327296:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.355339:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.355369:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.355372:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082327296:12345-192.168.202.105@tcp:104: Request processed in 42us (106us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.355377:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 277 00010000:00010000:2.0:1775764923.355392:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.355398:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.355405:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81902b9a00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.355409:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.355449:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.355454:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.355489:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.355497:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.355516:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.355525:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff55b3 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215576a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.355528:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066205696:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.355543:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.355546:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.355548:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.355953:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066205568:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.355961:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: --/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.355967:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: --/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.355972:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: --/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->217087] (req 0->217087) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.356427:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.356446:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.356457:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.356467:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.356476:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.356484:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.356499:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-lfsck.sh: write [0, 215320) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.356506:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356532:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356538:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000642df537[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356540:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000d469198 17ffffc0001021 2:0 ffff9d81906679a0 0 lru 00000080:00200000:2.0:1775764923.356544:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000aef20f75 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356546:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000642df537 00000080:00200000:2.0:1775764923.356547:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.356549:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356562:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356564:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dfb8a6e4[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356566:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d3b1469c 17ffffc0001001 3:0 ffff9d8190667540 1 lru 00000080:00200000:2.0:1775764923.356570:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000306785a3 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356571:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dfb8a6e4 00000080:00200000:2.0:1775764923.356572:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.356573:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356580:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356582:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f118adec[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356583:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e316d173 17ffffc0001001 3:0 ffff9d8190667000 2 lru 00000080:00200000:2.0:1775764923.356586:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000048899673 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356587:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f118adec 00000080:00200000:2.0:1775764923.356588:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.356589:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356595:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356598:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000607eb790[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356599:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000872c70fb 17ffffc0001001 3:0 ffff9d81906672a0 3 lru 00000080:00200000:2.0:1775764923.356603:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fbd0d281 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356604:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000607eb790 00000080:00200000:2.0:1775764923.356604:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.356605:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356613:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356615:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001cd6d042[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356616:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000755c11df 17ffffc0001001 3:0 ffff9d8190667ee0 4 lru 00000080:00200000:2.0:1775764923.356619:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008ea100ba 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356620:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001cd6d042 00000080:00200000:2.0:1775764923.356621:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.356622:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356635:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356637:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dace9afe[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356639:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001d837044 17ffffc0001001 3:0 ffff9d819031e7e0 5 lru 00000080:00200000:2.0:1775764923.356642:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006aa699ee 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356643:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dace9afe 00000080:00200000:2.0:1775764923.356644:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.356645:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356652:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356654:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000089702bad[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356655:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000052a23eb 17ffffc0001001 3:0 ffff9d819031e8c0 6 lru 00000080:00200000:2.0:1775764923.356658:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000032e842b5 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356660:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000089702bad 00000080:00200000:2.0:1775764923.356660:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.356661:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356668:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356670:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000055ce4de6[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356671:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000277d0b70 17ffffc0001001 3:0 ffff9d819031ec40 7 lru 00000080:00200000:2.0:1775764923.356674:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f1926245 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356675:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000055ce4de6 00000080:00200000:2.0:1775764923.356676:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.356677:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356684:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356686:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009ba795c5[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356688:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000921b0e3f 17ffffc0001001 3:0 ffff9d819031ea80 8 lru 00000080:00200000:2.0:1775764923.356691:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004fde72cb 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356692:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009ba795c5 00000080:00200000:2.0:1775764923.356692:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.356693:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356700:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356702:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000049f05ba8[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356703:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d0cc17dc 17ffffc0001001 3:0 ffff9d819031e380 9 lru 00000080:00200000:2.0:1775764923.356706:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001ab0147d 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356707:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000049f05ba8 00000080:00200000:2.0:1775764923.356708:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.356709:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356716:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356718:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f06dfcba[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356719:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000928ef05e 17ffffc0001001 3:0 ffff9d819031ee00 10 lru 00000080:00200000:2.0:1775764923.356723:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000842ce2a 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356724:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f06dfcba 00000080:00200000:2.0:1775764923.356724:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.356725:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356733:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356734:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d2c602e1[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356736:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006fbdfb9a 17ffffc0001001 3:0 ffff9d819031e1c0 11 lru 00000080:00200000:2.0:1775764923.356739:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b4fbcf10 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356740:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d2c602e1 00000080:00200000:2.0:1775764923.356740:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.356741:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356748:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356750:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005af39bef[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356751:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005f739e56 17ffffc0001001 3:0 ffff9d819031e700 12 lru 00000080:00200000:2.0:1775764923.356754:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000092cd97ee 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356756:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005af39bef 00000080:00200000:2.0:1775764923.356756:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.356757:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356764:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356765:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000069c19fc0[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356767:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000096720468 17ffffc0001001 3:0 ffff9d819031e620 13 lru 00000080:00200000:2.0:1775764923.356770:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f490946e 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356771:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000069c19fc0 00000080:00200000:2.0:1775764923.356771:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.356772:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356779:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356781:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a82e6781[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356782:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000064f1bed9 17ffffc0001001 3:0 ffff9d819031e460 14 lru 00000080:00200000:2.0:1775764923.356785:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f3e9872f 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356786:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a82e6781 00000080:00200000:2.0:1775764923.356787:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.356788:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356796:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356799:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000099dcdf[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356800:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000009e7aae0 17ffffc0001021 2:0 ffff9d819031e0e0 15 lru 00000080:00200000:2.0:1775764923.356803:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a531f117 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356804:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000099dcdf 00000080:00200000:2.0:1775764923.356804:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.356805:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356812:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356814:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d5b69d80[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356816:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000057dccb8e 17ffffc0001001 3:0 ffff9d819031eb60 16 lru 00000080:00200000:2.0:1775764923.356819:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000883851e9 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356820:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d5b69d80 00000080:00200000:2.0:1775764923.356821:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.356822:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356828:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356830:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005e753ef1[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356831:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008da1e141 17ffffc0001001 3:0 ffff9d819031e9a0 17 lru 00000080:00200000:2.0:1775764923.356835:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bf288746 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356836:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005e753ef1 00000080:00200000:2.0:1775764923.356836:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.356837:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356844:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356846:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008995917f[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356847:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000aaa51ea2 17ffffc0001001 3:0 ffff9d819031e540 18 lru 00000080:00200000:2.0:1775764923.356850:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000f15cc46 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356851:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008995917f 00000080:00200000:2.0:1775764923.356852:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.356853:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356860:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356862:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006a5017f3[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356864:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c827b025 17ffffc0001001 3:0 ffff9d819031e000 19 lru 00000080:00200000:2.0:1775764923.356867:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a0a1c63a 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356868:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006a5017f3 00000080:00200000:2.0:1775764923.356869:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.356869:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356894:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356897:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006e7075e3[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356898:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000041f50b52 17ffffc0001001 3:0 ffff9d819031e2a0 20 lru 00000080:00200000:2.0:1775764923.356902:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000016fe089f 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356903:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006e7075e3 00000080:00200000:2.0:1775764923.356904:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.356905:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356915:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356917:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f7b5fe83[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356919:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a01a8e96 17ffffc0001001 3:0 ffff9d819031eee0 21 lru 00000080:00200000:2.0:1775764923.356923:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000281471ab 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356925:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f7b5fe83 00000080:00200000:2.0:1775764923.356926:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.356927:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356936:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356939:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000045837de8[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356941:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000768ac3fc 17ffffc0001001 3:0 ffff9d819031ed20 22 lru 00000080:00200000:2.0:1775764923.356944:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001797302c 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356945:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000045837de8 00000080:00200000:2.0:1775764923.356946:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.356947:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.356960:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356963:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009c5f4154[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356964:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007953199a 17ffffc0001001 3:0 ffff9d8190e4c460 23 lru 00000080:00200000:2.0:1775764923.356968:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000055fe3fc 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356969:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009c5f4154 00000080:00200000:2.0:1775764923.356970:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.356971:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000100:00100000:1.0:1775764923.356976:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066205696:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.356979:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.356982:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a9a2eb56[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.356984:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002e983806 17ffffc0001001 3:0 ffff9d8190e4c0e0 24 lru 00000080:00200000:2.0:1775764923.356988:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000104dc279 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.356989:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a9a2eb56 00000080:00200000:2.0:1775764923.356990:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.356991:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357000:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357002:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f9bf4398[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357003:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003e81acbb 17ffffc0001001 3:0 ffff9d8190e4cb60 25 lru 00000080:00200000:2.0:1775764923.357006:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000590fcb6b 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357007:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f9bf4398 00000080:00200000:2.0:1775764923.357008:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.357009:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357016:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357017:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fb8add2b[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357019:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000078c97d0a 17ffffc0001001 3:0 ffff9d8190e4c9a0 26 lru 00000080:00200000:2.0:1775764923.357022:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000060137ae6 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357023:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fb8add2b 00000080:00200000:2.0:1775764923.357024:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.357025:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357031:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357033:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b564944d[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357034:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000de3182d5 17ffffc0001001 3:0 ffff9d8190e4c540 27 lru 00000080:00200000:2.0:1775764923.357037:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001824be15 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357038:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b564944d 00000080:00200000:2.0:1775764923.357039:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.357040:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357046:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357048:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000007f6755f[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357050:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000643c20c1 17ffffc0001001 3:0 ffff9d8190e4c000 28 lru 00000080:00200000:2.0:1775764923.357053:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d2f3c30a 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357054:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000007f6755f 00000080:00200000:2.0:1775764923.357055:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.357055:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357062:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357064:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002db4d8fb[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357065:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fe2f793e 17ffffc0001001 3:0 ffff9d8190e4c2a0 29 lru 00000080:00200000:2.0:1775764923.357068:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f022a5ac 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357069:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002db4d8fb 00000080:00200000:2.0:1775764923.357070:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.357070:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357080:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357082:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009197b3ff[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357083:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001ec96dc2 17ffffc0001021 2:0 ffff9d8190e4cee0 30 lru 00000080:00200000:2.0:1775764923.357087:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004d0edf8a 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357088:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009197b3ff 00000080:00200000:2.0:1775764923.357089:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.357090:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357096:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357098:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a3a1080b[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357100:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f07e70d8 17ffffc0001001 3:0 ffff9d8190e4cd20 31 lru 00000080:00200000:2.0:1775764923.357103:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a240c0df 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357104:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a3a1080b 00000080:00200000:2.0:1775764923.357105:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.357106:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357113:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357115:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000646f1f81[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357117:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000012b15fdd 17ffffc0001001 3:0 ffff9d8190e4c7e0 32 lru 00000080:00200000:2.0:1775764923.357120:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d1fa586a 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357128:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000646f1f81 00000080:00200000:2.0:1775764923.357129:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.357130:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357138:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357140:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007d35cb1e[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357141:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000080764d48 17ffffc0001001 3:0 ffff9d8190e4c8c0 33 lru 00000080:00200000:2.0:1775764923.357145:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000f89cf1d 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357147:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007d35cb1e 00000080:00200000:2.0:1775764923.357149:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.357150:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357162:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357166:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003d9fc4f8[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357168:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000712ce5cb 17ffffc0001001 3:0 ffff9d8190e4cc40 34 lru 00000080:00200000:2.0:1775764923.357187:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c0518c06 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357190:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003d9fc4f8 00000080:00200000:2.0:1775764923.357191:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.357193:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357208:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357212:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d8d74f5a[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357215:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000dc2bb77f 17ffffc0001001 3:0 ffff9d8190e4ca80 35 lru 00000080:00200000:2.0:1775764923.357220:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f6449529 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357222:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d8d74f5a 00000080:00200000:2.0:1775764923.357223:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.357225:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357236:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357240:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d131eac1[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357242:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000038a50ad2 17ffffc0001001 3:0 ffff9d8190e4c380 36 lru 00000080:00200000:2.0:1775764923.357247:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005459df14 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357249:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d131eac1 00000080:00200000:2.0:1775764923.357250:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.357252:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357264:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357268:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e26e5498[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357270:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000783805ad 17ffffc0001001 3:0 ffff9d8190e4ce00 37 lru 00000080:00200000:2.0:1775764923.357282:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000050efeb93 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357284:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e26e5498 00000080:00200000:2.0:1775764923.357285:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.357287:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357299:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357302:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a7d0ed90[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357303:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e3775575 17ffffc0001001 3:0 ffff9d8190e4c1c0 38 lru 00000080:00200000:2.0:1775764923.357307:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000095fa97bd 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357308:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a7d0ed90 00000080:00200000:2.0:1775764923.357309:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.357310:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357318:0:34105:0:(rw26.c:858:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357321:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000080bfa9bb[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357323:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a29fcdc2 17ffffc0001001 3:0 ffff9d8190e4c700 39 lru 00000080:00200000:2.0:1775764923.357326:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000408d952a 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357327:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000080bfa9bb 00000080:00200000:2.0:1775764923.357327:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1775764923.357328:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357340:0:34105:0:(rw26.c:858:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357343:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000287622ca[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357344:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002ba9b49f 17ffffc0001001 3:0 ffff9d8190e4c620 40 lru 00000080:00200000:2.0:1775764923.357347:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000017924fa5 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357348:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000287622ca 00000080:00200000:2.0:1775764923.357349:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1775764923.357350:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357362:0:34105:0:(rw26.c:858:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357364:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d2c0ef9e[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357365:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002676857e 17ffffc0001001 3:0 ffff9d81906a6a80 41 lru 00000080:00200000:2.0:1775764923.357368:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003e4852f4 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357370:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d2c0ef9e 00000080:00200000:2.0:1775764923.357370:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1775764923.357371:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357378:0:34105:0:(rw26.c:858:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357380:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000088d34d9[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357382:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000072d8c89a 17ffffc0001001 3:0 ffff9d81906a6380 42 lru 00000080:00200000:2.0:1775764923.357385:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bce598e3 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357386:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000088d34d9 00000080:00200000:2.0:1775764923.357387:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1775764923.357388:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357395:0:34105:0:(rw26.c:858:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357398:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bbf5f9d3[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357400:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b47ab632 17ffffc0001001 3:0 ffff9d81906a6e00 43 lru 00000080:00200000:2.0:1775764923.357405:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006c6233c6 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357407:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bbf5f9d3 00000080:00200000:2.0:1775764923.357408:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1775764923.357410:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357424:0:34105:0:(rw26.c:858:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357428:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d9f71c7a[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357431:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000df434055 17ffffc0001001 3:0 ffff9d81906a61c0 44 lru 00000080:00200000:2.0:1775764923.357436:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000060ac1826 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357438:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d9f71c7a 00000080:00200000:2.0:1775764923.357439:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1775764923.357441:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357459:0:34105:0:(rw26.c:858:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357463:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000076417dd4[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357465:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000027b2fc8f 17ffffc0001021 2:0 ffff9d81906a6700 45 lru 00000080:00200000:2.0:1775764923.357471:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000def3f894 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357473:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000076417dd4 00000080:00200000:2.0:1775764923.357474:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1775764923.357476:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357489:0:34105:0:(rw26.c:858:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357492:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008cd325aa[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357495:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bbe306b4 17ffffc0001001 3:0 ffff9d81906a6620 46 lru 00000080:00200000:2.0:1775764923.357500:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b0318197 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357502:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008cd325aa 00000080:00200000:2.0:1775764923.357503:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1775764923.357504:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357514:0:34105:0:(rw26.c:858:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357517:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e782ea8e[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357518:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000d36b969 17ffffc0001001 3:0 ffff9d81906a6460 47 lru 00000080:00200000:2.0:1775764923.357521:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005317f67a 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357522:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e782ea8e 00000080:00200000:2.0:1775764923.357523:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1775764923.357524:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357532:0:34105:0:(rw26.c:858:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357534:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000095123815[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357535:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cccfc8a6 17ffffc0001001 3:0 ffff9d81906a60e0 48 lru 00000080:00200000:2.0:1775764923.357539:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000004caa8f5 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357540:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000095123815 00000080:00200000:2.0:1775764923.357541:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1775764923.357541:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357548:0:34105:0:(rw26.c:858:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357550:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000787b988f[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357551:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000007b4aff8 17ffffc0001001 3:0 ffff9d81906a6b60 49 lru 00000080:00200000:2.0:1775764923.357554:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000008e8f7e2 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357555:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000787b988f 00000080:00200000:2.0:1775764923.357556:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1775764923.357557:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357563:0:34105:0:(rw26.c:858:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357565:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d58c745c[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357567:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005285d878 17ffffc0001001 3:0 ffff9d81906a69a0 50 lru 00000080:00200000:2.0:1775764923.357570:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008a1ce28b 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357571:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d58c745c 00000080:00200000:2.0:1775764923.357572:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1775764923.357573:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.357580:0:34105:0:(rw26.c:858:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:2.0:1775764923.357582:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000040130480[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357583:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c9473d1f 17ffffc0001001 3:0 ffff9d81906a6540 51 lru 00000080:00200000:2.0:1775764923.357586:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000348f61f2 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357587:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000040130480 00000080:00200000:2.0:1775764923.357588:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1775764923.357589:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 52 of 0 to 2328 bytes 00000080:00200000:2.0:1775764923.357597:0:34105:0:(rw26.c:858:ll_write_end()) pos 212992, len 2328, copied 2328 00000080:00200000:2.0:1775764923.357599:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a69ab692[3 000000005f4550fb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.357601:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005321543e 17ffffc0001001 3:0 ffff9d81906a6000 52 lru 00000080:00200000:2.0:1775764923.357604:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ccae1cb0 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 000000009452db5f 000000001997ee6f > 3< 0 0 > 4< 0 0 8 2535424 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.357605:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a69ab692 00000080:00200000:2.0:1775764923.357606:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1775764923.357608:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 53, from 0, to 2328 prio 0 00000008:00010000:2.0:1775764923.357617:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357626:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357631:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357636:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357644:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357648:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357652:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357656:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357660:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357664:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357668:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357672:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357676:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357681:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357685:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.357693:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f0278, count 15, dirtied 15 00000008:00010000:2.0:1775764923.357697:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357701:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357705:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357710:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357714:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357718:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357725:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357730:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357734:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357738:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357742:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357746:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357750:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357754:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357758:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.357765:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f0278, count 15, dirtied 15 00000008:00010000:2.0:1775764923.357767:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357772:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357776:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357780:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357784:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357788:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357792:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357796:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357803:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357807:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357811:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357816:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357820:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357824:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357828:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.357833:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f0278, count 15, dirtied 15 00000008:00010000:2.0:1775764923.357836:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357843:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357847:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357851:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357855:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357859:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357863:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.357867:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000004809a715 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.357872:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f0278, count 8, dirtied 8 00000080:00200000:2.0:1775764923.357890:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 53 pages 215320 bytes, tot: 215320 00000080:00200000:2.0:1775764923.357892:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-lfsck.sh: write bytes 0, result: 215320 00000020:00010000:2.0:1775764923.357898:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.357900:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.357902:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ceab0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.357905:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.357910:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.357917:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.357919:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ceab0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.357923:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 215320 bytes 00000080:00200000:2.0:1775764923.357925:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 52] 00000080:00200000:2.0:1775764923.357934:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3e:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.357937:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-lfsck.sh: 2 io complete with rc: 0, result: 215320, restart: 0 00000080:00200000:2.0:1775764923.357940:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3e:0x0] updating i_size 215320 i_blocks 0 00000080:00200000:2.0:1775764923.357942:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 215320 00000080:00200008:2.0:1775764923.357957:0:34105:0:(file.c:447:ll_file_release()) START file sanity-lfsck.sh:[0x240004281:0x3e:0x0](ffff9d81843f00c8), flags 100001 00000002:00100000:2.0:1775764923.357973:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906f8700 x1862023066205312/t38654705881(38654705881) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.357980:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.357983:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f8000 x1862023066205952/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.357991:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066205952:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.358021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9b40 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.358641:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066205952:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.358654:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906f8700 x1862023066205312/t38654705881(38654705881) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.358680:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.358686:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.358691:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.358697:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.358702:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.358704:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.358706:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.358708:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.358710:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.358714:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.358719:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.358722:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.358724:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.358726:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.358727:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.358729:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-lfsck.sh, flags=8 00000080:00200008:2.0:1775764923.358732:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-lfsck.sh:[0x240004281:0x3e:0x0](ffff9d81843f00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.358734:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3e:0x0](ffff9d81843f00c8),name=sanity-lfsck.sh 00000002:00010000:2.0:1775764923.358739:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3e:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.358742:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.358747:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.358750:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.358752:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3e:0x0](ffff9d81843f00c8) 00000080:00010000:2.0:1775764923.358753:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3e:0x0](ffff9d81843f00c8) for lock 0xeef44110eff55ac 00000080:00010000:2.0:1775764923.358755:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55ac from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.358756:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.358760:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.358773:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.358777:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3e:0x0] 00000008:00010000:2.0:1775764923.358785:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.358787:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ceab0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.358792:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.358798:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.358803:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.358805:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.358810:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000bc9759bf/0xeef44110eff55ba lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->217087) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155771 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.358814:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.358817:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3e:0x0] updating i_size 215320 i_blocks 0 00000020:00010000:2.0:1775764923.358820:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.358822:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.358824:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ceab0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.358827:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.358830:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ceab0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.358841:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.358896:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.358905:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.358912:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.358918:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.358923:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.358926:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.358928:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.358931:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.358933:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.358941:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.358946:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.358948:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.358950:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.358952:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.358953:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.358958:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-lnet.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.358960:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lnet.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.358965:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lnet.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.358978:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.358982:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.358986:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.358994:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066206080:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.359022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9b40 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.359803:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066206080:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.359812:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359819:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359825:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359830:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359837:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359841:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8c00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.359843:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359847:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359851:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359854:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.359856:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.359859:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff55c1 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.359866:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f9c00 x1862023066206080/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.359872:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.359910:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.359931:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.359938:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.359944:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.359950:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.359954:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.359956:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.359958:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.359960:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.359962:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.359966:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.359971:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.359974:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.359976:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.359977:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.359979:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.359982:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-lnet.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.359985:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lnet.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.359989:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lnet.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.360001:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360008:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360012:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.360019:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066206208:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.360046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9b40 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.360776:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066206208:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.360782:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360786:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360790:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360793:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360797:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360800:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8a00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.360801:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360805:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360808:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360811:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.360814:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360817:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff55c8 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.360822:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f9c00 x1862023066206208/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.360828:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.360834:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.360860:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.360864:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.360868:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.360873:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.360894:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.360897:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.360899:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.360901:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.360903:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.360911:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.360916:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.360918:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.360921:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.360922:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.360924:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.360928:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.360931:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-lnet.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da900, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.360934:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lnet.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.360943:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lnet.sh,[0x240004281:0x3f:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.360965:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.360971:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.360978:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.360983:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f9c00 x1862023066206464/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.360991:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.361002:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066206464:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.361061:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9b40 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.362105:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066206464:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.362112:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362120:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362131:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 4/0,1 mode: --/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362137:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362141:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362147:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362151:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362155:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.362160:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906f9c00 x1862023066206464/t38654705883(38654705883) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.362166:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f9c00 x1862023066206464/t38654705883(38654705883) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.362185:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.362191:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362196:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/63 (0 0) 00000080:00200000:2.0:1775764923.362198:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.362202:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-lnet.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.362212:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x3f:0x0](ffff9d81843f6c08) 00000080:00200000:2.0:1775764923.362263:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x3f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.362274:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x3f:0x0](ffff9d81843f6c08) 00000080:00010000:2.0:1775764923.362276:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3f:0x0](ffff9d81843f6c08) for lock 0xeef44110eff55cf 00000080:00010000:2.0:1775764923.362282:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3f:0x0](ffff9d81843f6c08) 00000080:00200008:2.0:1775764923.362287:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-lnet.sh:[0x240004281:0x3f:0x0](ffff9d81843f6c08), flags 100301 00000002:00100000:2.0:1775764923.362300:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f9c00 x1862023066206464/t38654705883(38654705883) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.362323:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55cf from it ffff9d8190e6e240 00010000:00010000:2.0:1775764923.362325:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362331:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.362346:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x3f:0x0](ffff9d81843f6c08), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.362353:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-lnet.sh:[0x240004281:0x3f:0x0](ffff9d81843f6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.362355:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3f:0x0](ffff9d81843f6c08),name=sanity-lnet.sh 00000002:00010000:2.0:1775764923.362362:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3f:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.362366:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.362371:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.362375:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.362378:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3f:0x0](ffff9d81843f6c08) 00000080:00010000:2.0:1775764923.362379:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3f:0x0](ffff9d81843f6c08) for lock 0xeef44110eff55cf 00000080:00010000:2.0:1775764923.362381:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55cf from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.362383:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.362387:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.362399:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.362402:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3f:0x0] 00000008:00010000:2.0:1775764923.362411:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.362412:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce330, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.362426:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.362432:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.362454:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1775764923.363201:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.363210:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.363217:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.363226:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.363233:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.363239:0:10947:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.363247:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.363252:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.363256:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.363266:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.363271:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3f:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.363276:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000100:00100000:1.0:1775764923.363277:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066206592:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.363282:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.363286:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce330 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.363289:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.363292:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce330 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.363306:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.363366:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-lnet.sh:[0x240004281:0x3f:0x0], ppos: 0, count: 184445 00000080:00200000:2.0:1775764923.363376:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-lnet.sh: write ppos: 0, bytes: 184445 00000080:00200000:2.0:1775764923.363379:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-lnet.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.363382:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000788040c6[0x110, 1, [0x240004281:0x3f:0x0] hash] 00000020:00200000:2.0:1775764923.363383:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 184445) 0 0 00000080:00200000:2.0:1775764923.363385:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3f:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.363389:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 45] 00020000:00200000:2.0:1775764923.363391:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 184445) 184445, 184445 00020000:00200000:2.0:1775764923.363393:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.363402:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 45] 00000008:00010000:2.0:1775764923.363406:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.363407:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce330, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.363413:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/544 (0 188415) 00010000:00010000:2.0:1775764923.363432:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.363437:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->188415] (req 0->188415) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.363442:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->188415] (req 0->188415) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.363451:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fa300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066206720:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.363475:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.364063:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082327552 00000100:00100000:1.0:1775764923.364067:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082327552 00000100:00100000:1.0:1775764923.364090:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 279 00000100:00100000:1.0:1775764923.364095:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082327552:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.364103:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.364140:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.364144:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082327552:12345-192.168.202.105@tcp:104: Request processed in 50us (108us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.364150:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 279 00010000:00010000:2.0:1775764923.364158:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.364163:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.364178:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8188fb8e00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.364183:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.364206:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.364210:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.364243:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.364250:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.364270:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.364278:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff55d6 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321557c5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.364289:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.364291:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.364293:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.364306:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775764923.364318:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066206848:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.364721:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fa300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066206720:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.364728:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->188415] (req 0->188415) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.364734:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->188415] (req 0->188415) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.364739:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->188415] (req 0->188415) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.365118:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.365125:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.365142:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.365149:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.365155:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.365160:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.365182:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-lnet.sh: write [0, 184445) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.365188:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365205:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365209:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009a979d86[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365211:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000458ebe73 17ffffc0001001 3:0 ffff9d81906a62a0 0 lru 00000080:00200000:2.0:1775764923.365216:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000415b2c5 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365217:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009a979d86 00000080:00200000:2.0:1775764923.365218:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.365220:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365230:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365233:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f28d8f7e[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365235:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002c821f40 17ffffc0001001 3:0 ffff9d81906a6ee0 1 lru 00000080:00200000:2.0:1775764923.365239:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ac8279f0 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365241:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f28d8f7e 00000080:00200000:2.0:1775764923.365242:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.365244:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365257:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365261:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008dd9a091[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365264:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000076ec6c06 17ffffc0001001 3:0 ffff9d81906a6d20 2 lru 00000080:00200000:2.0:1775764923.365270:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000aec14f63 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365272:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008dd9a091 00000080:00200000:2.0:1775764923.365274:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.365276:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365289:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365293:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000030f5d44[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365296:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000009cfaac5 17ffffc0001001 3:0 ffff9d81906a67e0 3 lru 00000080:00200000:2.0:1775764923.365301:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cb326b5a 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365303:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000030f5d44 00000080:00200000:2.0:1775764923.365304:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.365307:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365322:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365326:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003cf04ce9[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365329:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000802bb75f 17ffffc0001001 3:0 ffff9d81906a68c0 4 lru 00000080:00200000:2.0:1775764923.365334:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000955036d4 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365336:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003cf04ce9 00000080:00200000:2.0:1775764923.365337:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.365339:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365352:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365357:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004c09c722[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365359:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004cfb15b0 17ffffc0001001 3:0 ffff9d81906a6c40 5 lru 00000080:00200000:2.0:1775764923.365364:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dab61a41 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365367:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004c09c722 00000080:00200000:2.0:1775764923.365368:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.365369:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365385:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365387:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f5f743f6[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365389:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000047f8905f 17ffffc0001001 3:0 ffff9d8190f35e00 6 lru 00000080:00200000:2.0:1775764923.365392:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009e350b2b 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365394:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f5f743f6 00000080:00200000:2.0:1775764923.365394:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.365395:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365406:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365409:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d43334f5[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365410:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004f674b3e 17ffffc0001021 2:0 ffff9d8190f351c0 7 lru 00000080:00200000:2.0:1775764923.365413:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003b75a93a 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365415:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d43334f5 00000080:00200000:2.0:1775764923.365416:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.365416:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365424:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365425:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009e03249c[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365427:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004de9098c 17ffffc0001001 3:0 ffff9d8190f35700 8 lru 00000080:00200000:2.0:1775764923.365430:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008a2cb9e9 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365431:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009e03249c 00000080:00200000:2.0:1775764923.365432:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.365433:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365440:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365441:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000074b62ca8[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365443:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004f63bd5d 17ffffc0001001 3:0 ffff9d8190f35620 9 lru 00000080:00200000:2.0:1775764923.365446:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b61c04df 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365447:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000074b62ca8 00000080:00200000:2.0:1775764923.365447:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.365448:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365455:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365457:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008b9e3442[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365459:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000034bff36a 17ffffc0001001 3:0 ffff9d8190f35460 10 lru 00000080:00200000:2.0:1775764923.365462:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000953c1fb1 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365463:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008b9e3442 00000080:00200000:2.0:1775764923.365464:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.365464:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365471:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365473:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009217415e[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365474:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000081a4dd6c 17ffffc0001001 3:0 ffff9d8190f350e0 11 lru 00000080:00200000:2.0:1775764923.365477:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d76dc565 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365479:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009217415e 00000080:00200000:2.0:1775764923.365479:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.365480:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365487:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365489:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000007beef7d[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365491:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002f8998c6 17ffffc0001001 3:0 ffff9d8190f35b60 12 lru 00000080:00200000:2.0:1775764923.365494:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000017eab11 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365495:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000007beef7d 00000080:00200000:2.0:1775764923.365496:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.365496:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365503:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365505:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c59b391f[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365507:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000804fa347 17ffffc0001001 3:0 ffff9d8190f359a0 13 lru 00000080:00200000:2.0:1775764923.365510:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a0c0fcf8 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365511:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c59b391f 00000080:00200000:2.0:1775764923.365511:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.365512:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365524:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365527:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000afa74489[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365529:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000297dd4ed 17ffffc0001001 3:0 ffff9d8190f35540 14 lru 00000080:00200000:2.0:1775764923.365534:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002b1ced09 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365535:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000afa74489 00000080:00200000:2.0:1775764923.365536:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.365538:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365552:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365556:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002e492638[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365559:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009b14c6b1 17ffffc0001001 3:0 ffff9d8190f35000 15 lru 00000080:00200000:2.0:1775764923.365564:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e2b01ac0 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365566:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002e492638 00000080:00200000:2.0:1775764923.365567:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.365569:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365582:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365586:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000782d846b[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365589:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000708ba653 17ffffc0001001 3:0 ffff9d8190f352a0 16 lru 00000080:00200000:2.0:1775764923.365595:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f4c8f069 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365597:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000782d846b 00000080:00200000:2.0:1775764923.365599:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.365600:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365612:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365615:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002d694885[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365617:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b70c4cb8 17ffffc0001001 3:0 ffff9d8190f35ee0 17 lru 00000080:00200000:2.0:1775764923.365620:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008dc3fa4b 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365622:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002d694885 00000080:00200000:2.0:1775764923.365623:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.365624:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365633:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365635:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000072321fb6[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365637:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000016f6502 17ffffc0001001 3:0 ffff9d8190f35d20 18 lru 00000080:00200000:2.0:1775764923.365641:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005b10e758 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365642:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000072321fb6 00000080:00200000:2.0:1775764923.365643:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.365644:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365651:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365654:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000018253511[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365656:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004b2724fd 17ffffc0001001 3:0 ffff9d8190f357e0 19 lru 00000080:00200000:2.0:1775764923.365660:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d29161c0 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365662:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000018253511 00000080:00200000:2.0:1775764923.365663:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000100:00100000:1.0:1775764923.365665:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066206848:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.365665:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365676:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365679:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000083ff2836[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365681:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007e014dbc 17ffffc0001001 3:0 ffff9d8190f358c0 20 lru 00000080:00200000:2.0:1775764923.365684:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001342b818 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365686:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000083ff2836 00000080:00200000:2.0:1775764923.365687:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.365688:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365695:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365697:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000060f9d8c0[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365698:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000342f69a3 17ffffc0001001 3:0 ffff9d8190f35c40 21 lru 00000080:00200000:2.0:1775764923.365701:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000046a8d1b6 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365703:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000060f9d8c0 00000080:00200000:2.0:1775764923.365703:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.365704:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365713:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365715:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009807578b[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365717:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006142567f 17ffffc0001021 2:0 ffff9d8190f35a80 22 lru 00000080:00200000:2.0:1775764923.365720:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000056a50684 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365721:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009807578b 00000080:00200000:2.0:1775764923.365722:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.365723:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365729:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365731:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000023f18456[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365732:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000030dd8a5e 17ffffc0001001 3:0 ffff9d8190f35380 23 lru 00000080:00200000:2.0:1775764923.365735:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bb0d2e1b 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365736:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000023f18456 00000080:00200000:2.0:1775764923.365737:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.365738:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365750:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365752:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008ce5b065[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365754:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b5e99aa4 17ffffc0001001 3:0 ffff9d81904c1620 24 lru 00000080:00200000:2.0:1775764923.365757:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fd54108d 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365759:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008ce5b065 00000080:00200000:2.0:1775764923.365759:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.365760:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365767:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365769:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dd1ac488[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365770:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ff20ae15 17ffffc0001001 3:0 ffff9d81904c1460 25 lru 00000080:00200000:2.0:1775764923.365773:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003903c87b 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365774:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dd1ac488 00000080:00200000:2.0:1775764923.365775:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.365776:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365782:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365784:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e65b27a8[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365786:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008fb2f8b1 17ffffc0001001 3:0 ffff9d81904c10e0 26 lru 00000080:00200000:2.0:1775764923.365789:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d8858f87 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365791:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e65b27a8 00000080:00200000:2.0:1775764923.365791:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.365792:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365799:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365801:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000000aa2498[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365802:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d5061a64 17ffffc0001001 3:0 ffff9d81904c1b60 27 lru 00000080:00200000:2.0:1775764923.365805:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a9399c52 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365806:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000000aa2498 00000080:00200000:2.0:1775764923.365807:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.365808:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365816:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365818:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000030fa7f8f[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365819:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005dcf760a 17ffffc0001001 3:0 ffff9d81904c19a0 28 lru 00000080:00200000:2.0:1775764923.365822:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000025bbb1ab 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365823:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000030fa7f8f 00000080:00200000:2.0:1775764923.365823:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.365824:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365831:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365833:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000061e25eb0[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365834:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000034acbb3 17ffffc0001001 3:0 ffff9d81904c1540 29 lru 00000080:00200000:2.0:1775764923.365837:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000def5d9f4 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365838:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000061e25eb0 00000080:00200000:2.0:1775764923.365838:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.365839:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365846:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365848:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000636d92da[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365849:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e749ebbd 17ffffc0001001 3:0 ffff9d81904c1000 30 lru 00000080:00200000:2.0:1775764923.365852:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d0f3fa8c 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365853:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000636d92da 00000080:00200000:2.0:1775764923.365854:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.365855:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365861:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365863:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005393c159[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365865:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a2b82e75 17ffffc0001001 3:0 ffff9d81904c12a0 31 lru 00000080:00200000:2.0:1775764923.365868:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a1106d3f 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365869:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005393c159 00000080:00200000:2.0:1775764923.365869:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.365870:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365894:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365897:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007aab4f8d[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365898:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009fd5771e 17ffffc0001001 3:0 ffff9d81904c1ee0 32 lru 00000080:00200000:2.0:1775764923.365901:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ac37487f 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365902:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007aab4f8d 00000080:00200000:2.0:1775764923.365903:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.365904:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365912:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365914:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000076cf7954[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365916:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d38eb770 17ffffc0001001 3:0 ffff9d81904c1d20 33 lru 00000080:00200000:2.0:1775764923.365919:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004d6d6f4f 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365920:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000076cf7954 00000080:00200000:2.0:1775764923.365921:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.365922:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365928:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365930:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005e4e68dc[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365931:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000579eabc7 17ffffc0001001 3:0 ffff9d81904c17e0 34 lru 00000080:00200000:2.0:1775764923.365934:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cbf33f30 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365935:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005e4e68dc 00000080:00200000:2.0:1775764923.365936:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.365937:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365945:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365947:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000085bf9037[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365948:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002d42ecd1 17ffffc0001001 3:0 ffff9d81904c18c0 35 lru 00000080:00200000:2.0:1775764923.365951:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004284b222 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365953:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000085bf9037 00000080:00200000:2.0:1775764923.365954:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.365954:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365961:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365963:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005591a235[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365964:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009d1a0af4 17ffffc0001001 3:0 ffff9d81904c1c40 36 lru 00000080:00200000:2.0:1775764923.365968:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000004669db2 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365969:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005591a235 00000080:00200000:2.0:1775764923.365970:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.365971:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365979:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365981:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d91bbec1[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365983:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b262ab9c 17ffffc0001021 2:0 ffff9d81904c1a80 37 lru 00000080:00200000:2.0:1775764923.365985:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fc2ffc03 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.365987:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d91bbec1 00000080:00200000:2.0:1775764923.365987:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.365988:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.365995:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.365997:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000704f6faa[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.365999:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001921d4e9 17ffffc0001001 3:0 ffff9d81904c1380 38 lru 00000080:00200000:2.0:1775764923.366002:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f01ba53d 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.366003:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000704f6faa 00000080:00200000:2.0:1775764923.366003:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.366004:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.366010:0:34105:0:(rw26.c:858:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.366012:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000071773aa7[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.366014:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000079c8dd05 17ffffc0001001 3:0 ffff9d81904c1e00 39 lru 00000080:00200000:2.0:1775764923.366017:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e61769e2 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.366018:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000071773aa7 00000080:00200000:2.0:1775764923.366018:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1775764923.366019:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.366026:0:34105:0:(rw26.c:858:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.366028:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000087f3f298[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.366029:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000753b2642 17ffffc0001001 3:0 ffff9d81904c11c0 40 lru 00000080:00200000:2.0:1775764923.366032:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004d05d2ea 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.366033:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000087f3f298 00000080:00200000:2.0:1775764923.366034:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1775764923.366035:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.366041:0:34105:0:(rw26.c:858:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1775764923.366043:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ef08de80[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.366044:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000050ad8f2f 17ffffc0001001 3:0 ffff9d81904c1700 41 lru 00000080:00200000:2.0:1775764923.366047:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bcb3b63f 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.366049:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ef08de80 00000080:00200000:2.0:1775764923.366049:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1775764923.366050:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.366063:0:34105:0:(rw26.c:858:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1775764923.366065:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000029e36418[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.366066:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b2961f70 17ffffc0001001 3:0 ffff9d818775ce00 42 lru 00000080:00200000:2.0:1775764923.366070:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000066950acd 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.366071:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000029e36418 00000080:00200000:2.0:1775764923.366072:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1775764923.366072:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.366079:0:34105:0:(rw26.c:858:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1775764923.366081:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000118ebde7[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.366082:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000092c3a367 17ffffc0001001 3:0 ffff9d818775c1c0 43 lru 00000080:00200000:2.0:1775764923.366085:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a6c73b34 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.366086:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000118ebde7 00000080:00200000:2.0:1775764923.366087:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1775764923.366088:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.366094:0:34105:0:(rw26.c:858:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1775764923.366096:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c12a714a[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.366098:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d6c11505 17ffffc0001001 3:0 ffff9d818775c700 44 lru 00000080:00200000:2.0:1775764923.366101:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000082be8919 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.366103:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c12a714a 00000080:00200000:2.0:1775764923.366103:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1775764923.366104:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 45 of 0 to 125 bytes 00000080:00200000:2.0:1775764923.366113:0:34105:0:(rw26.c:858:ll_write_end()) pos 184320, len 125, copied 125 00000080:00200000:2.0:1775764923.366115:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000000ea5f67[3 00000000de7159d1 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.366117:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000082844fd7 17ffffc0001001 3:0 ffff9d818775c620 45 lru 00000080:00200000:2.0:1775764923.366120:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000046400f5 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 00000000409b8422 00000000ef56f9f6 > 3< 0 0 > 4< 0 0 8 3612672 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.366121:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000000ea5f67 00000080:00200000:2.0:1775764923.366122:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1775764923.366123:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 46, from 0, to 125 prio 0 00000008:00010000:2.0:1775764923.366131:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366139:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366144:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366149:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366153:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366157:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366161:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366166:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366184:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366189:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366194:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366198:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366203:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366207:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366212:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.366219:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f6db8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.366223:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366228:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366232:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366236:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366244:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366251:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366259:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366267:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366275:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366289:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366298:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366305:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366312:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366317:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366321:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.366328:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f6db8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.366331:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366335:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366343:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366350:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366358:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366365:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366373:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366380:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366388:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366397:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366405:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366413:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366421:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366427:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.366431:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.366438:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f6db8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.366441:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000c379fae1 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.366445:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f6db8, count 1, dirtied 1 00000080:00200000:2.0:1775764923.366447:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 46 pages 184445 bytes, tot: 184445 00000080:00200000:2.0:1775764923.366449:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-lnet.sh: write bytes 0, result: 184445 00000020:00010000:2.0:1775764923.366454:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.366456:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.366457:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce330 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.366460:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.366465:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.366471:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.366473:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce330 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.366476:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 184445 bytes 00000080:00200000:2.0:1775764923.366478:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 45] 00000080:00200000:2.0:1775764923.366487:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3f:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.366489:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-lnet.sh: 2 io complete with rc: 0, result: 184445, restart: 0 00000080:00200000:2.0:1775764923.366493:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3f:0x0] updating i_size 184445 i_blocks 0 00000080:00200000:2.0:1775764923.366495:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 184445 00000080:00200008:2.0:1775764923.366508:0:34105:0:(file.c:447:ll_file_release()) START file sanity-lnet.sh:[0x240004281:0x3f:0x0](ffff9d81843f6c08), flags 100001 00000002:00100000:2.0:1775764923.366524:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906f9c00 x1862023066206464/t38654705883(38654705883) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.366530:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.366533:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fa300 x1862023066207104/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.366541:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fa300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207104:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.366570:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.367348:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fa300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207104:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.367361:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906f9c00 x1862023066206464/t38654705883(38654705883) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.367388:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.367396:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.367403:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.367416:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.367422:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.367426:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.367428:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.367432:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.367434:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.367441:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.367449:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.367453:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.367456:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.367458:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.367459:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.367461:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-lnet.sh, flags=8 00000080:00200008:2.0:1775764923.367464:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-lnet.sh:[0x240004281:0x3f:0x0](ffff9d81843f6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.367467:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x3f:0x0](ffff9d81843f6c08),name=sanity-lnet.sh 00000002:00010000:2.0:1775764923.367473:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x3f:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.367477:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.367482:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.367487:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.367489:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x3f:0x0](ffff9d81843f6c08) 00000080:00010000:2.0:1775764923.367490:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x3f:0x0](ffff9d81843f6c08) for lock 0xeef44110eff55cf 00000080:00010000:2.0:1775764923.367493:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55cf from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.367494:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.367499:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.367512:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x3f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.367515:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x3f:0x0] 00000008:00010000:2.0:1775764923.367524:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.367525:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce330, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.367530:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.367535:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.367540:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.367542:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.367546:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004ef7d4ae/0xeef44110eff55dd lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x220:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->188415) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321557cc expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.367555:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.367557:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x3f:0x0] updating i_size 184445 i_blocks 0 00000020:00010000:2.0:1775764923.367560:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.367561:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.367563:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce330 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.367565:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.367566:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce330 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.367573:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x3f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.367595:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.367600:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.367604:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.367609:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.367613:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.367615:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.367617:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.367619:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.367621:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.367625:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.367633:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.367635:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.367636:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.367638:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.367639:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.367644:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.367646:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.367650:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lsnapshot.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.367663:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.367667:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.367671:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.367678:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fb480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207232:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.367706:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.368557:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fb480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207232:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.368563:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368567:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368571:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368575:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368579:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368582:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583b800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.368584:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368587:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368590:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368593:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.368596:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368600:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b800/0xeef44110eff55e4 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.368606:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fb480 x1862023066207232/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.368611:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.368618:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.368632:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.368637:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.368640:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.368646:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.368650:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.368652:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.368653:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.368655:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.368657:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.368661:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.368666:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.368668:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.368670:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.368671:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.368673:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.368676:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.368678:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.368682:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lsnapshot.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.368693:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368697:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.368703:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.368711:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fb480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207360:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.368737:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.369572:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fb480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207360:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.369578:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369583:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369586:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369590:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369593:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369596:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583a600 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.369598:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369601:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369604:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369608:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.369610:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369613:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff55eb lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.369619:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fb480 x1862023066207360/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.369625:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.369631:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.369664:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.369671:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.369678:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.369686:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.369692:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.369696:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.369698:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.369701:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.369705:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.369712:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.369720:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.369724:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.369727:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.369730:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.369732:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.369739:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.369742:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da000, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.369747:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.369757:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-lsnapshot.sh,[0x240004281:0x40:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.369772:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.369776:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.369781:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.369784:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fb480 x1862023066207616/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.369789:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.369796:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fb480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207616:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.369846:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8184cd9f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.370928:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fb480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207616:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.370935:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.370954:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x40:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.370958:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.370963:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.370968:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.370974:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.370978:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.370982:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.370986:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fb480 x1862023066207616/t38654705885(38654705885) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.370992:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fb480 x1862023066207616/t38654705885(38654705885) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.370997:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.371003:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.371007:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/64 (0 0) 00000080:00200000:2.0:1775764923.371010:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.371013:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-lsnapshot.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.371022:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x40:0x0](ffff9d81843f1148) 00000080:00200000:2.0:1775764923.371061:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x40:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.371066:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x40:0x0](ffff9d81843f1148) 00000080:00010000:2.0:1775764923.371068:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x40:0x0](ffff9d81843f1148) for lock 0xeef44110eff55f2 00000080:00010000:2.0:1775764923.371071:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x40:0x0](ffff9d81843f1148) 00000080:00200008:2.0:1775764923.371075:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-lsnapshot.sh:[0x240004281:0x40:0x0](ffff9d81843f1148), flags 100301 00000002:00100000:2.0:1775764923.371083:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fb480 x1862023066207616/t38654705885(38654705885) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.371099:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55f2 from it ffff9d8190e6e240 00010000:00010000:2.0:1775764923.371102:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.371107:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.371118:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x40:0x0](ffff9d81843f1148), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.371127:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-lsnapshot.sh:[0x240004281:0x40:0x0](ffff9d81843f1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.371131:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x40:0x0](ffff9d81843f1148),name=sanity-lsnapshot.sh 00000002:00010000:2.0:1775764923.371140:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x40:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.371148:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.371156:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.371163:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.371166:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x40:0x0](ffff9d81843f1148) 00000080:00010000:2.0:1775764923.371183:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x40:0x0](ffff9d81843f1148) for lock 0xeef44110eff55f2 00000080:00010000:2.0:1775764923.371188:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55f2 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.371191:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.371198:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.371217:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x40:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.371222:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x40:0x0] 00000008:00010000:2.0:1775764923.371231:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.371232:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce530, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.371252:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fcb5bcd/0xeef44110eff55f9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.371260:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fcb5bcd/0xeef44110eff55f9 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.371282:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1775764923.371956:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x40:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.371962:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.371965:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.371968:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce530 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.371974:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.371978:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce530 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.371992:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x40:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.372011:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-lsnapshot.sh:[0x240004281:0x40:0x0], ppos: 0, count: 10934 00000080:00200000:2.0:1775764923.372020:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-lsnapshot.sh: write ppos: 0, bytes: 10934 00000080:00200000:2.0:1775764923.372022:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-lsnapshot.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.372025:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@0000000003b2eca2[0x110, 1, [0x240004281:0x40:0x0] hash] 00000020:00200000:2.0:1775764923.372026:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 10934) 0 0 00000080:00200000:2.0:1775764923.372028:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x40:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.372032:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 2] 00020000:00200000:2.0:1775764923.372034:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 10934) 10934, 10934 00020000:00200000:2.0:1775764923.372036:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.372044:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 2] 00000008:00010000:2.0:1775764923.372048:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.372050:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce530, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.372054:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/545 (0 12287) 00010000:00010000:2.0:1775764923.372066:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.372072:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->12287] (req 0->12287) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.372076:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->12287] (req 0->12287) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.372085:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8a80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207872:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.372108:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1780 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.372745:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.372753:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082327808 00000100:00100000:1.0:1775764923.372756:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082327808 00000100:00100000:1.0:1775764923.372776:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 281 00000100:00100000:1.0:1775764923.372782:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796cc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082327808:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.372789:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fcb5bcd/0xeef44110eff55f9 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155820 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.372825:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.372831:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796cc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082327808:12345-192.168.202.105@tcp:104: Request processed in 49us (125us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.372838:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 281 00010000:00010000:2.0:1775764923.372845:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.372851:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fcb5bcd/0xeef44110eff55f9 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155820 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.372859:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583a800 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.372864:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fcb5bcd/0xeef44110eff55f9 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155820 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.372919:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.372925:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.372957:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fcb5bcd/0xeef44110eff55f9 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155820 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.372962:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.372978:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fcb5bcd/0xeef44110eff55f9 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155820 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.372986:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fcb5bcd/0xeef44110eff55f9 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155820 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.372998:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.373000:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.373002:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.373008:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066208000:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.373432:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8a80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066207872:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.373444:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->12287] (req 0->12287) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.373457:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->12287] (req 0->12287) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.373465:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->12287] (req 0->12287) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.373794:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.373810:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.373822:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.373828:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.373834:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.373839:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.373848:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-lsnapshot.sh: write [0, 10934) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.373854:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.373870:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.373888:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006c453ea6[3 000000000e44457b 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.373891:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004b374109 17ffffc0001001 3:0 ffff9d818775c460 0 lru 00000080:00200000:2.0:1775764923.373896:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a4f4fcc2 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 000000006a30a9d0 > 3< 0 0 > 4< 0 0 8 2293760 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.373897:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006c453ea6 00000080:00200000:2.0:1775764923.373898:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.373900:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.373913:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.373916:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009509a529[3 000000000e44457b 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.373918:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c1bd1f7e 17ffffc0001001 3:0 ffff9d818775c0e0 1 lru 00000080:00200000:2.0:1775764923.373922:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000055b654be 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 000000006a30a9d0 > 3< 0 0 > 4< 0 0 8 2293760 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.373924:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009509a529 00000080:00200000:2.0:1775764923.373926:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.373928:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 2742 bytes 00000080:00200000:2.0:1775764923.373944:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 2742, copied 2742 00000080:00200000:2.0:1775764923.373948:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000d4fd32a[3 000000000e44457b 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.373951:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f708bf07 17ffffc0001001 3:0 ffff9d818775cb60 2 lru 00000080:00200000:2.0:1775764923.373956:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000eb1da4ab 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000009452db5f 000000006a30a9d0 > 3< 0 0 > 4< 0 0 8 2293760 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.373958:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000d4fd32a 00000080:00200000:2.0:1775764923.373959:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.373962:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 3, from 0, to 2742 prio 0 00000008:00010000:2.0:1775764923.373972:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000025e7376 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.373984:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000025e7376 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.373993:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000025e7376 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.374008:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f12f8, count 3, dirtied 3 00000080:00200000:2.0:1775764923.374011:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 3 pages 10934 bytes, tot: 10934 00000080:00200000:2.0:1775764923.374014:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-lsnapshot.sh: write bytes 0, result: 10934 00000020:00010000:2.0:1775764923.374019:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.374021:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.374023:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce530 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.374025:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.374031:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.374036:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.374038:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce530 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.374041:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 10934 bytes 00000080:00200000:2.0:1775764923.374043:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 2] 00000080:00200000:2.0:1775764923.374051:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x40:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.374053:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-lsnapshot.sh: 2 io complete with rc: 0, result: 10934, restart: 0 00000080:00200000:2.0:1775764923.374057:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x40:0x0] updating i_size 10934 i_blocks 0 00000080:00200000:2.0:1775764923.374059:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 10934 00000080:00200008:2.0:1775764923.374071:0:34105:0:(file.c:447:ll_file_release()) START file sanity-lsnapshot.sh:[0x240004281:0x40:0x0](ffff9d81843f1148), flags 100001 00000002:00100000:2.0:1775764923.374086:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fb480 x1862023066207616/t38654705885(38654705885) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.374092:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.374095:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f8a80 x1862023066208256/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.374107:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8a80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066208256:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.374139:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1780 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.374380:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066208000:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.374744:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8a80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066208256:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.374753:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fb480 x1862023066207616/t38654705885(38654705885) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.374772:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.374778:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.374783:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.374789:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.374794:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.374797:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.374800:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.374804:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.374806:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.374813:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.374822:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.374826:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.374830:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.374833:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.374835:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.374838:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-lsnapshot.sh, flags=8 00000080:00200008:2.0:1775764923.374843:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-lsnapshot.sh:[0x240004281:0x40:0x0](ffff9d81843f1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.374847:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x40:0x0](ffff9d81843f1148),name=sanity-lsnapshot.sh 00000002:00010000:2.0:1775764923.374855:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x40:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.374861:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.374869:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.374873:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.374890:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x40:0x0](ffff9d81843f1148) 00000080:00010000:2.0:1775764923.374892:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x40:0x0](ffff9d81843f1148) for lock 0xeef44110eff55f2 00000080:00010000:2.0:1775764923.374895:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff55f2 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.374897:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.374902:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.374916:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x40:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.374919:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x40:0x0] 00000008:00010000:2.0:1775764923.374927:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.374929:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce530, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.374934:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.374943:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.374948:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.374950:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.374954:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000002cc7a634/0xeef44110eff5600 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155827 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.374959:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.374961:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x40:0x0] updating i_size 10934 i_blocks 0 00000020:00010000:2.0:1775764923.374963:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.374965:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.374966:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce530 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.374969:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.374970:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce530 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.374978:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x40:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.375006:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.375014:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.375021:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.375029:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.375040:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.375044:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.375047:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.375050:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.375053:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.375061:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.375068:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.375072:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.375074:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.375076:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.375077:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.375082:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanityn.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.375084:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanityn.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.375090:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanityn.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.375103:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375107:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375111:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.375118:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906faa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066208384:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.375146:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1780 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.375910:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906faa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066208384:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.375916:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375921:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375925:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375928:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375931:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375934:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583b600 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.375936:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375939:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375943:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375946:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.375948:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.375954:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5607 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.375960:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906faa00 x1862023066208384/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.375966:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.375972:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.375987:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.375992:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.375995:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.376001:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.376004:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.376006:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.376008:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.376010:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.376011:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.376016:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.376020:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.376022:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.376025:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.376026:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.376028:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.376032:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanityn.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.376033:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanityn.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.376038:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanityn.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.376048:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376052:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376056:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.376063:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906faa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066208512:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.376090:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1780 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.376936:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906faa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066208512:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.376942:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376947:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376951:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376954:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376961:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376964:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583ba00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.376966:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376969:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376972:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376975:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.376977:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.376980:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff560e lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.376987:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906faa00 x1862023066208512/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.376993:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.376999:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.377023:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.377028:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.377032:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.377040:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.377044:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.377046:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.377047:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.377049:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.377051:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.377055:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.377060:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.377062:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.377064:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.377065:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.377066:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.377071:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.377073:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanityn.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da180, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.377076:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanityn.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.377083:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanityn.sh,[0x240004281:0x41:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.377096:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.377100:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.377104:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.377108:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906faa00 x1862023066208768/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.377113:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.377120:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906faa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066208768:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.377165:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1780 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.378195:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906faa00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066208768:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.378202:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378210:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x41:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378214:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378219:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378224:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378229:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378237:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378240:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.378245:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906faa00 x1862023066208768/t38654705887(38654705887) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.378251:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906faa00 x1862023066208768/t38654705887(38654705887) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.378256:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.378261:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378266:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/65 (0 0) 00000080:00200000:2.0:1775764923.378268:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.378272:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanityn.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.378280:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x41:0x0](ffff9d81843f1988) 00000080:00200000:2.0:1775764923.378347:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x41:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.378357:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x41:0x0](ffff9d81843f1988) 00000080:00010000:2.0:1775764923.378360:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x41:0x0](ffff9d81843f1988) for lock 0xeef44110eff5615 00000080:00010000:2.0:1775764923.378366:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x41:0x0](ffff9d81843f1988) 00000080:00200008:2.0:1775764923.378372:0:34105:0:(file.c:1057:ll_file_open()) START file sanityn.sh:[0x240004281:0x41:0x0](ffff9d81843f1988), flags 100301 00000002:00100000:2.0:1775764923.378384:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906faa00 x1862023066208768/t38654705887(38654705887) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.378407:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5615 from it ffff9d8190e6e240 00010000:00010000:2.0:1775764923.378411:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378419:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.378430:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x41:0x0](ffff9d81843f1988), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.378438:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanityn.sh:[0x240004281:0x41:0x0](ffff9d81843f1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.378440:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x41:0x0](ffff9d81843f1988),name=sanityn.sh 00000002:00010000:2.0:1775764923.378447:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x41:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.378452:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.378457:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.378461:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.378463:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x41:0x0](ffff9d81843f1988) 00000080:00010000:2.0:1775764923.378465:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x41:0x0](ffff9d81843f1988) for lock 0xeef44110eff5615 00000080:00010000:2.0:1775764923.378467:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5615 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.378469:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.378473:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.378494:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x41:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.378498:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x41:0x0] 00000008:00010000:2.0:1775764923.378506:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.378507:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce2b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.378522:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.378527:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.378542:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1775764923.379206:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.379216:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.379223:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.379229:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.379235:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.379250:0:10947:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.379256:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.379261:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.379266:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.379274:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.379282:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066208896:192.168.202.105@tcp:101:cp.0 00000080:00200000:2.0:1775764923.379282:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x41:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.379286:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.379289:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.379292:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce2b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.379294:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.379296:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce2b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.379309:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x41:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.379391:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanityn.sh:[0x240004281:0x41:0x0], ppos: 0, count: 218297 00000080:00200000:2.0:1775764923.379406:0:34105:0:(file.c:2058:ll_file_io_generic()) sanityn.sh: write ppos: 0, bytes: 218297 00000080:00200000:2.0:1775764923.379409:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanityn.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.379413:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000e5887c47[0x110, 1, [0x240004281:0x41:0x0] hash] 00000020:00200000:2.0:1775764923.379414:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 218297) 0 0 00000080:00200000:2.0:1775764923.379417:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x41:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.379422:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 53] 00020000:00200000:2.0:1775764923.379426:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 218297) 218297, 218297 00020000:00200000:2.0:1775764923.379428:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.379440:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 53] 00000008:00010000:2.0:1775764923.379444:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.379446:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce2b0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.379450:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/545 (0 221183) 00010000:00010000:2.0:1775764923.379462:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.379468:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->221183] (req 0->221183) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.379472:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->221183] (req 0->221183) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.379480:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9880 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209024:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.379504:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc16c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.380116:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082328064 00000100:00100000:1.0:1775764923.380121:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082328064 00000100:00100000:1.0:1775764923.380149:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 283 00000100:00100000:1.0:1775764923.380155:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082328064:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.380161:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.380204:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.380208:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082328064:12345-192.168.202.105@tcp:104: Request processed in 54us (130us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.380214:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 283 00010000:00010000:2.0:1775764923.380224:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.380229:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.380235:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583be00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.380239:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.380269:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x41:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.380275:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x41:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.380300:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.380305:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.380322:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.380329:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000089a6d247/0xeef44110eff561c lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215587b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.380342:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00010000:00010000:2.0:1775764923.380342:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.380345:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.380349:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.380354:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066209152:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.380712:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9880 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209024:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.380720:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->221183] (req 0->221183) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.380725:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->221183] (req 0->221183) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.380730:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->221183] (req 0->221183) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.381158:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.381183:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.381195:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.381205:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.381212:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.381217:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.381229:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanityn.sh: write [0, 218297) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.381234:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381254:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381260:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000383f8509[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381263:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e31e425d 17ffffc0001001 3:0 ffff9d818775c9a0 0 lru 00000080:00200000:2.0:1775764923.381269:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007fcad2b5 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381271:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000383f8509 00000080:00200000:2.0:1775764923.381272:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.381274:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381299:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381306:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000db03f4fc[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381308:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a9aa31cf 17ffffc0001001 3:0 ffff9d818775c540 1 lru 00000080:00200000:2.0:1775764923.381312:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002e7b3894 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381313:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000db03f4fc 00000080:00200000:2.0:1775764923.381314:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.381315:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381325:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381328:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008da420ee[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381329:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000557139b1 17ffffc0001001 3:0 ffff9d818775c000 2 lru 00000080:00200000:2.0:1775764923.381333:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000340e984 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381334:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008da420ee 00000080:00200000:2.0:1775764923.381335:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.381336:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381346:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381348:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a952d431[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381350:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000024d00947 17ffffc0001021 2:0 ffff9d818775c2a0 3 lru 00000080:00200000:2.0:1775764923.381353:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c2ee0069 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381354:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a952d431 00000080:00200000:2.0:1775764923.381355:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.381356:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381363:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381365:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e7a6bfa6[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381366:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000036a83a53 17ffffc0001001 3:0 ffff9d818775cee0 4 lru 00000080:00200000:2.0:1775764923.381369:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bead2c9f 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381370:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e7a6bfa6 00000080:00200000:2.0:1775764923.381371:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.381371:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381378:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381381:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000aad840eb[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381384:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000006ec03e9 17ffffc0001001 3:0 ffff9d818775cd20 5 lru 00000080:00200000:2.0:1775764923.381388:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007c82b45f 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381390:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000aad840eb 00000080:00200000:2.0:1775764923.381391:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.381392:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381404:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381408:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000086d0d289[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381410:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009f29c119 17ffffc0001001 3:0 ffff9d818775c7e0 6 lru 00000080:00200000:2.0:1775764923.381415:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000631075fb 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381417:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000086d0d289 00000080:00200000:2.0:1775764923.381418:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.381420:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381434:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381437:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000adbb2779[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381439:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008ac5f276 17ffffc0001001 3:0 ffff9d818775c8c0 7 lru 00000080:00200000:2.0:1775764923.381442:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000015209ae 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381444:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000adbb2779 00000080:00200000:2.0:1775764923.381444:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.381446:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381454:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381456:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fab0e816[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381457:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b3722020 17ffffc0001001 3:0 ffff9d818775cc40 8 lru 00000080:00200000:2.0:1775764923.381460:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d5c4da1d 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381462:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fab0e816 00000080:00200000:2.0:1775764923.381462:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.381463:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381470:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381472:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f23ff04b[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381473:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005c401126 17ffffc0001001 3:0 ffff9d818775ca80 9 lru 00000080:00200000:2.0:1775764923.381476:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a6417976 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381478:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f23ff04b 00000080:00200000:2.0:1775764923.381478:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.381479:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381486:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381487:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bbbc9d66[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381489:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c1e8efad 17ffffc0001001 3:0 ffff9d818775c380 10 lru 00000080:00200000:2.0:1775764923.381492:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000632fefa0 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381493:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bbbc9d66 00000080:00200000:2.0:1775764923.381493:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.381494:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381506:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381508:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000018ea5125[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381509:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d14a3e13 17ffffc0001001 3:0 ffff9d81863392a0 11 lru 00000080:00200000:2.0:1775764923.381513:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000059eb52fa 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381514:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000018ea5125 00000080:00200000:2.0:1775764923.381515:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.381516:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381522:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381524:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c85f4405[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381526:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c150fd2c 17ffffc0001001 3:0 ffff9d8186339ee0 12 lru 00000080:00200000:2.0:1775764923.381529:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006a6f1de3 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381530:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c85f4405 00000080:00200000:2.0:1775764923.381531:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.381532:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381538:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381540:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f0731037[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381541:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008f2561eb 17ffffc0001001 3:0 ffff9d8186339d20 13 lru 00000080:00200000:2.0:1775764923.381544:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000546df04e 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381545:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f0731037 00000080:00200000:2.0:1775764923.381546:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.381547:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381554:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381556:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000eef83d60[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381558:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000d1e5eb5 17ffffc0001001 3:0 ffff9d81863397e0 14 lru 00000080:00200000:2.0:1775764923.381561:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000451f2ba7 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381562:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000eef83d60 00000080:00200000:2.0:1775764923.381562:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.381563:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381569:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381571:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000058c8a556[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381573:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000090a55bef 17ffffc0001001 3:0 ffff9d81863398c0 15 lru 00000080:00200000:2.0:1775764923.381576:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000057f28448 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381578:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000058c8a556 00000080:00200000:2.0:1775764923.381578:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.381579:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381585:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381587:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000084f4a5d3[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381589:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004551d8c8 17ffffc0001001 3:0 ffff9d8186339c40 16 lru 00000080:00200000:2.0:1775764923.381592:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e313e8c2 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381593:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000084f4a5d3 00000080:00200000:2.0:1775764923.381594:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.381595:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381602:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381605:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b7e44dd3[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381607:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000da8f923e 17ffffc0001001 3:0 ffff9d8186339a80 17 lru 00000080:00200000:2.0:1775764923.381610:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fb646e0a 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381611:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b7e44dd3 00000080:00200000:2.0:1775764923.381612:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.381613:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381623:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381626:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000004c6d950[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381627:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000781ad743 17ffffc0001021 2:0 ffff9d8186339380 18 lru 00000080:00200000:2.0:1775764923.381631:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007a163bf2 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381633:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000004c6d950 00000080:00200000:2.0:1775764923.381634:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.381635:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381643:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381646:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c9c761d1[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381648:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000bfabfbf 17ffffc0001001 3:0 ffff9d8186339e00 19 lru 00000080:00200000:2.0:1775764923.381651:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ef91b36b 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381653:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c9c761d1 00000080:00200000:2.0:1775764923.381654:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.381655:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381662:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381664:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bb724e66[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381666:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000093a44033 17ffffc0001001 3:0 ffff9d81863391c0 20 lru 00000080:00200000:2.0:1775764923.381670:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000be88e253 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381672:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bb724e66 00000080:00200000:2.0:1775764923.381673:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.381674:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381683:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000100:00100000:1.0:1775764923.381685:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066209152:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.381688:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006cecdb18[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381693:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003981ffa7 17ffffc0001001 3:0 ffff9d8186339700 21 lru 00000080:00200000:2.0:1775764923.381697:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000948e47b8 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381698:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006cecdb18 00000080:00200000:2.0:1775764923.381699:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.381700:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381708:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381711:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c9fb39e2[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381712:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005b416bd4 17ffffc0001001 3:0 ffff9d8186339620 22 lru 00000080:00200000:2.0:1775764923.381715:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009166c1e7 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381716:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c9fb39e2 00000080:00200000:2.0:1775764923.381717:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.381718:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381724:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381726:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000056e1b683[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381727:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000052f0e117 17ffffc0001001 3:0 ffff9d8186339460 23 lru 00000080:00200000:2.0:1775764923.381730:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000a6358da 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381731:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000056e1b683 00000080:00200000:2.0:1775764923.381732:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.381733:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381739:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381741:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005707fe7a[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381742:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000840e98c4 17ffffc0001001 3:0 ffff9d81863390e0 24 lru 00000080:00200000:2.0:1775764923.381745:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f5f09c17 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381746:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005707fe7a 00000080:00200000:2.0:1775764923.381747:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.381748:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381755:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381757:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c2aa17ae[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381758:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000030455390 17ffffc0001001 3:0 ffff9d8186339b60 25 lru 00000080:00200000:2.0:1775764923.381761:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004b33e4c1 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381762:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c2aa17ae 00000080:00200000:2.0:1775764923.381763:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.381764:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381770:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381772:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000022f5a758[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381773:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000000c2e038 17ffffc0001001 3:0 ffff9d81863399a0 26 lru 00000080:00200000:2.0:1775764923.381776:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000be290b67 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381777:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000022f5a758 00000080:00200000:2.0:1775764923.381778:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.381778:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381785:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381786:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002add2151[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381788:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008f27a15f 17ffffc0001001 3:0 ffff9d8186339540 27 lru 00000080:00200000:2.0:1775764923.381791:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000011b3acef 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381792:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002add2151 00000080:00200000:2.0:1775764923.381792:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.381793:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381800:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381802:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007c01a89f[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381803:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000061e3263 17ffffc0001001 3:0 ffff9d8186339000 28 lru 00000080:00200000:2.0:1775764923.381806:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f94b57ae 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381807:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007c01a89f 00000080:00200000:2.0:1775764923.381808:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.381809:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381820:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381822:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000346bdb67[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381823:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a9749d0b 17ffffc0001001 3:0 ffff9d81846ebe00 29 lru 00000080:00200000:2.0:1775764923.381827:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000324bf1ce 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381828:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000346bdb67 00000080:00200000:2.0:1775764923.381829:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.381830:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381837:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381838:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e2a39f20[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381840:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000c9dfc19 17ffffc0001001 3:0 ffff9d81846eb1c0 30 lru 00000080:00200000:2.0:1775764923.381843:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dde51d84 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381844:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e2a39f20 00000080:00200000:2.0:1775764923.381845:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.381846:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381854:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381856:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003b11d937[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381857:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b727f6bd 17ffffc0001001 3:0 ffff9d81846eb700 31 lru 00000080:00200000:2.0:1775764923.381860:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009bf6dd28 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381861:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003b11d937 00000080:00200000:2.0:1775764923.381862:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.381863:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381869:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381871:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f0056bf8[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381873:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fc292074 17ffffc0001001 3:0 ffff9d81846eb620 32 lru 00000080:00200000:2.0:1775764923.381890:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c62addaf 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381891:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f0056bf8 00000080:00200000:2.0:1775764923.381892:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.381893:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381904:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381907:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a8c53df8[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381909:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008df0248c 17ffffc0001021 2:0 ffff9d81846eb460 33 lru 00000080:00200000:2.0:1775764923.381912:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004db668f3 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381913:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a8c53df8 00000080:00200000:2.0:1775764923.381913:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.381915:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381921:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381923:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000002531bff[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381925:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000044ade671 17ffffc0001001 3:0 ffff9d81846eb0e0 34 lru 00000080:00200000:2.0:1775764923.381928:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000663b672a 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381929:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000002531bff 00000080:00200000:2.0:1775764923.381930:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.381931:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381938:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381940:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000097c4222a[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381941:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005c173f50 17ffffc0001001 3:0 ffff9d81846ebb60 35 lru 00000080:00200000:2.0:1775764923.381944:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c9f6cd37 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381946:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000097c4222a 00000080:00200000:2.0:1775764923.381946:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.381947:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381954:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381956:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bfdf3b41[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381957:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d3783682 17ffffc0001001 3:0 ffff9d81846eb9a0 36 lru 00000080:00200000:2.0:1775764923.381960:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ad3170a4 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381962:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bfdf3b41 00000080:00200000:2.0:1775764923.381962:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.381963:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381969:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381971:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000013b0089d[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381973:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003c051e26 17ffffc0001001 3:0 ffff9d81846eb540 37 lru 00000080:00200000:2.0:1775764923.381976:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000012bacb04 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381977:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000013b0089d 00000080:00200000:2.0:1775764923.381977:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.381978:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.381985:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.381987:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000e6a8389[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.381988:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c49b520c 17ffffc0001001 3:0 ffff9d81846eb000 38 lru 00000080:00200000:2.0:1775764923.381992:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dc1769c5 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.381993:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000e6a8389 00000080:00200000:2.0:1775764923.381993:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.381994:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382001:0:34105:0:(rw26.c:858:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382003:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000488d03cf[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382004:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c9edc3fb 17ffffc0001001 3:0 ffff9d81846eb2a0 39 lru 00000080:00200000:2.0:1775764923.382007:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000042d3d49 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382008:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000488d03cf 00000080:00200000:2.0:1775764923.382009:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1775764923.382009:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382016:0:34105:0:(rw26.c:858:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382018:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000719e28cc[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382019:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fdf765ce 17ffffc0001001 3:0 ffff9d81846ebee0 40 lru 00000080:00200000:2.0:1775764923.382022:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000700c06df 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382023:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000719e28cc 00000080:00200000:2.0:1775764923.382024:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1775764923.382024:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382031:0:34105:0:(rw26.c:858:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382032:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b7924964[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382034:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006f2e2da2 17ffffc0001001 3:0 ffff9d81846ebd20 41 lru 00000080:00200000:2.0:1775764923.382037:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002067d99f 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382038:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b7924964 00000080:00200000:2.0:1775764923.382038:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1775764923.382039:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382055:0:34105:0:(rw26.c:858:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382057:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d7977e38[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382059:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002fc453fb 17ffffc0001001 3:0 ffff9d81846eb7e0 42 lru 00000080:00200000:2.0:1775764923.382061:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000040c74831 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382063:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d7977e38 00000080:00200000:2.0:1775764923.382063:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1775764923.382064:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382072:0:34105:0:(rw26.c:858:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382074:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000aafa8d85[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382075:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005aa96597 17ffffc0001001 3:0 ffff9d81846eb8c0 43 lru 00000080:00200000:2.0:1775764923.382078:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000015c3dbff 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382080:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000aafa8d85 00000080:00200000:2.0:1775764923.382080:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1775764923.382081:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382087:0:34105:0:(rw26.c:858:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382089:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d557a24b[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382090:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ba1fc095 17ffffc0001001 3:0 ffff9d81846ebc40 44 lru 00000080:00200000:2.0:1775764923.382093:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009682790e 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382094:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d557a24b 00000080:00200000:2.0:1775764923.382095:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1775764923.382096:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382102:0:34105:0:(rw26.c:858:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382104:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006a595ec7[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382105:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008fb609d3 17ffffc0001001 3:0 ffff9d81846eba80 45 lru 00000080:00200000:2.0:1775764923.382108:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f7edfcb2 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382109:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006a595ec7 00000080:00200000:2.0:1775764923.382110:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1775764923.382110:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382117:0:34105:0:(rw26.c:858:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382118:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004f499628[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382120:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000005b1e6e7 17ffffc0001001 3:0 ffff9d81846eb380 46 lru 00000080:00200000:2.0:1775764923.382123:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000025c5375e 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382124:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004f499628 00000080:00200000:2.0:1775764923.382124:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1775764923.382125:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382136:0:34105:0:(rw26.c:858:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382138:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000798062b4[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382140:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000033426b25 17ffffc0001001 3:0 ffff9d81877228c0 47 lru 00000080:00200000:2.0:1775764923.382143:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000007023f9 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382145:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000798062b4 00000080:00200000:2.0:1775764923.382145:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1775764923.382147:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382160:0:34105:0:(rw26.c:858:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382164:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bfb59764[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382167:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003d1f4bdb 17ffffc0001021 2:0 ffff9d8187722c40 48 lru 00000080:00200000:2.0:1775764923.382183:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a2f08ca8 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382185:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bfb59764 00000080:00200000:2.0:1775764923.382187:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1775764923.382188:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382205:0:34105:0:(rw26.c:858:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382209:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bb466331[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382211:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001240c626 17ffffc0001001 3:0 ffff9d8187722a80 49 lru 00000080:00200000:2.0:1775764923.382217:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000810988d5 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382219:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bb466331 00000080:00200000:2.0:1775764923.382220:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1775764923.382222:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382233:0:34105:0:(rw26.c:858:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382235:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bde8f3d7[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382237:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f314a89f 17ffffc0001001 3:0 ffff9d8187722380 50 lru 00000080:00200000:2.0:1775764923.382242:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ff2cab0c 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382244:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bde8f3d7 00000080:00200000:2.0:1775764923.382245:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1775764923.382247:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382260:0:34105:0:(rw26.c:858:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382264:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b2e049ca[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382267:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000064c02e53 17ffffc0001001 3:0 ffff9d8187722e00 51 lru 00000080:00200000:2.0:1775764923.382272:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f42be7e8 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382274:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b2e049ca 00000080:00200000:2.0:1775764923.382275:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1775764923.382277:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 52 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.382289:0:34105:0:(rw26.c:858:ll_write_end()) pos 212992, len 4096, copied 4096 00000080:00200000:2.0:1775764923.382293:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ae0de889[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382296:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001c0ed2a4 17ffffc0001001 3:0 ffff9d81877221c0 52 lru 00000080:00200000:2.0:1775764923.382301:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000318be0dd 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382303:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ae0de889 00000080:00200000:2.0:1775764923.382304:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1775764923.382306:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 53 of 0 to 1209 bytes 00000080:00200000:2.0:1775764923.382319:0:34105:0:(rw26.c:858:ll_write_end()) pos 217088, len 1209, copied 1209 00000080:00200000:2.0:1775764923.382322:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000eb85e6a5[3 00000000f4c0d05d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.382324:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000036a2d248 17ffffc0001001 3:0 ffff9d8187722700 53 lru 00000080:00200000:2.0:1775764923.382327:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e8d20d67 53: 1< 0 - - > 2< 217088 0 0 0x0 0x0 | 00000000409b8422 00000000838b6305 > 3< 0 0 > 4< 0 0 8 3399680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.382328:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000eb85e6a5 00000080:00200000:2.0:1775764923.382329:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1775764923.382330:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 54, from 0, to 1209 prio 0 00000008:00010000:2.0:1775764923.382339:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382346:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382351:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382356:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382360:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382367:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382372:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382376:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382381:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382388:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382395:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382402:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382411:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382420:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382428:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.382440:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f1b38, count 15, dirtied 15 00000008:00010000:2.0:1775764923.382443:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382448:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382453:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382457:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382461:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382465:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382469:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382476:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382480:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382484:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382488:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382492:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382496:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382500:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382504:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.382510:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f1b38, count 15, dirtied 15 00000008:00010000:2.0:1775764923.382513:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382518:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382522:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382526:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382530:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382534:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382538:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382542:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382546:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382553:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382558:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382562:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382566:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382570:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382574:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.382579:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f1b38, count 15, dirtied 15 00000008:00010000:2.0:1775764923.382582:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382586:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382593:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382597:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382601:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382605:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382609:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382613:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.382617:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000d5545307 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.382622:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f1b38, count 9, dirtied 9 00000080:00200000:2.0:1775764923.382624:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 54 pages 218297 bytes, tot: 218297 00000080:00200000:2.0:1775764923.382626:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanityn.sh: write bytes 0, result: 218297 00000020:00010000:2.0:1775764923.382631:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.382632:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.382634:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce2b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.382637:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.382642:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.382648:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.382649:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce2b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.382652:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 218297 bytes 00000080:00200000:2.0:1775764923.382654:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 53] 00000080:00200000:2.0:1775764923.382663:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x41:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.382665:0:34105:0:(file.c:2206:ll_file_io_generic()) sanityn.sh: 2 io complete with rc: 0, result: 218297, restart: 0 00000080:00200000:2.0:1775764923.382668:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x41:0x0] updating i_size 218297 i_blocks 0 00000080:00200000:2.0:1775764923.382670:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 218297 00000080:00200008:2.0:1775764923.382683:0:34105:0:(file.c:447:ll_file_release()) START file sanityn.sh:[0x240004281:0x41:0x0](ffff9d81843f1988), flags 100001 00000002:00100000:2.0:1775764923.382698:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906faa00 x1862023066208768/t38654705887(38654705887) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.382704:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.382707:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f9880 x1862023066209408/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.382715:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9880 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209408:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.382744:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc16c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.383425:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9880 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209408:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.383437:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906faa00 x1862023066208768/t38654705887(38654705887) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.383461:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.383467:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.383473:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.383481:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.383488:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.383491:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.383494:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.383498:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.383501:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.383508:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.383517:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.383521:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.383525:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.383527:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.383529:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.383531:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanityn.sh, flags=8 00000080:00200008:2.0:1775764923.383534:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanityn.sh:[0x240004281:0x41:0x0](ffff9d81843f1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.383537:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x41:0x0](ffff9d81843f1988),name=sanityn.sh 00000002:00010000:2.0:1775764923.383542:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x41:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.383547:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.383552:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.383556:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.383559:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x41:0x0](ffff9d81843f1988) 00000080:00010000:2.0:1775764923.383560:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x41:0x0](ffff9d81843f1988) for lock 0xeef44110eff5615 00000080:00010000:2.0:1775764923.383563:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5615 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.383564:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.383570:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.383590:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x41:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.383595:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x41:0x0] 00000008:00010000:2.0:1775764923.383608:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.383610:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce2b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.383617:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.383627:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.383635:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.383637:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.383645:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000049123180/0xeef44110eff5623 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x221:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->221183) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155882 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.383653:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.383658:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x41:0x0] updating i_size 218297 i_blocks 0 00000020:00010000:2.0:1775764923.383660:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.383662:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.383664:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce2b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.383666:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.383667:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce2b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.383676:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x41:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.383698:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.383704:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.383709:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.383714:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.383720:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.383723:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.383726:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.383729:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.383732:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.383743:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.383752:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.383755:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.383759:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.383762:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.383764:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.383771:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-pcc.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.383774:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-pcc.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.383782:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-pcc.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.383801:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.383808:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.383815:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.383828:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209536:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.383865:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc16c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.384653:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209536:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.384659:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384664:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384668:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384671:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384675:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384678:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583b200 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.384680:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384683:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384686:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384689:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.384691:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384695:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff562a lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.384702:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f8380 x1862023066209536/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.384708:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.384714:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.384731:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.384735:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.384739:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.384744:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.384748:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.384750:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.384751:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.384753:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.384755:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.384759:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.384764:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.384766:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.384768:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.384770:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.384771:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.384775:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-pcc.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.384777:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-pcc.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.384781:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-pcc.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.384792:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384799:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.384803:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.384810:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209664:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.384842:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc16c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.385626:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209664:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.385635:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385642:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385647:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385653:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385659:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385664:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583ac00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.385667:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385670:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385674:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385678:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.385680:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385683:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ac00/0xeef44110eff5631 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.385691:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906f8380 x1862023066209664/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.385697:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.385703:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.385730:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.385735:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.385738:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.385743:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.385748:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.385750:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.385751:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.385753:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.385755:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.385763:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.385768:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.385770:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.385772:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.385773:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.385775:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.385779:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.385782:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-pcc.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851dac00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.385785:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-pcc.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.385792:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-pcc.sh,[0x240004281:0x42:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.385805:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.385809:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.385814:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.385817:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f8380 x1862023066209920/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.385822:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.385829:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209920:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.385896:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc16c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.386833:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f8380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066209920:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.386839:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.386848:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x42:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.386852:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.386857:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.386861:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.386866:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.386870:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.386899:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.386905:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906f8380 x1862023066209920/t38654705889(38654705889) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.386912:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f8380 x1862023066209920/t38654705889(38654705889) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.386918:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.386924:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.386929:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/66 (0 0) 00000080:00200000:2.0:1775764923.386931:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.386935:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-pcc.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.386944:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x42:0x0](ffff9d81843f2a08) 00000080:00200000:2.0:1775764923.386983:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x42:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.386990:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x42:0x0](ffff9d81843f2a08) 00000080:00010000:2.0:1775764923.386992:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x42:0x0](ffff9d81843f2a08) for lock 0xeef44110eff5638 00000080:00010000:2.0:1775764923.386996:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x42:0x0](ffff9d81843f2a08) 00000080:00200008:2.0:1775764923.387001:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-pcc.sh:[0x240004281:0x42:0x0](ffff9d81843f2a08), flags 100301 00000002:00100000:2.0:1775764923.387009:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906f8380 x1862023066209920/t38654705889(38654705889) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.387025:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5638 from it ffff9d8190e6e240 00010000:00010000:2.0:1775764923.387028:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.387033:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.387040:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x42:0x0](ffff9d81843f2a08), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.387047:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-pcc.sh:[0x240004281:0x42:0x0](ffff9d81843f2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.387050:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x42:0x0](ffff9d81843f2a08),name=sanity-pcc.sh 00000002:00010000:2.0:1775764923.387057:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x42:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.387061:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.387066:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.387070:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.387072:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x42:0x0](ffff9d81843f2a08) 00000080:00010000:2.0:1775764923.387073:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x42:0x0](ffff9d81843f2a08) for lock 0xeef44110eff5638 00000080:00010000:2.0:1775764923.387075:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5638 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.387077:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.387081:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.387093:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x42:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.387096:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x42:0x0] 00000008:00010000:2.0:1775764923.387104:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.387105:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cec30, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.387118:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.387123:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.387142:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.387190:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066210048:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.387824:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.387843:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.387853:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.387862:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.387871:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.387905:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.387922:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.387930:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.387936:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x42:0x0] updating i_size 0 i_blocks 0 00010000:00010000:1.0:1775764923.387939:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1775764923.387939:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.387944:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.387948:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cec30 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.387950:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.387951:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cec30 (ffff9d8191255518/1) at cl_lock_fini():72 00010000:00010000:1.0:1775764923.387960:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.387965:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x42:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1775764923.387980:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066210048:192.168.202.105@tcp:101:cp.0 00000080:00200008:2.0:1775764923.388035:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-pcc.sh:[0x240004281:0x42:0x0], ppos: 0, count: 159631 00000080:00200000:2.0:1775764923.388050:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-pcc.sh: write ppos: 0, bytes: 159631 00000080:00200000:2.0:1775764923.388053:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-pcc.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.388056:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000972d570c[0x110, 1, [0x240004281:0x42:0x0] hash] 00000020:00200000:2.0:1775764923.388058:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 159631) 0 0 00000080:00200000:2.0:1775764923.388059:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x42:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.388063:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 38] 00020000:00200000:2.0:1775764923.388066:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 159631) 159631, 159631 00020000:00200000:2.0:1775764923.388068:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.388077:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 38] 00000008:00010000:2.0:1775764923.388081:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.388083:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cec30, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.388087:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/546 (0 159743) 00010000:00010000:2.0:1775764923.388099:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.388104:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->159743] (req 0->159743) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.388108:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->159743] (req 0->159743) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.388117:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066210176:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.388142:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1900 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.388868:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082328320 00000100:00100000:1.0:1775764923.388874:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082328320 00000100:00100000:1.0:1775764923.388923:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 285 00000100:00100000:1.0:1775764923.388932:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082328320:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.388940:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.388976:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.388982:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082328320:12345-192.168.202.105@tcp:104: Request processed in 53us (201us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.388989:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 285 00010000:00010000:2.0:1775764923.389002:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.389009:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.389017:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583a600 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.389021:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.389059:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x42:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.389065:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.389096:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 2/0,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.389104:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.389124:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 1/0,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.389133:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d10bf07a/0xeef44110eff563f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321558d6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.389145:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.389149:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066210304:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.389149:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.389152:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.389523:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066210176:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.389534:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->159743] (req 0->159743) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.389543:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->159743] (req 0->159743) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.389551:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->159743] (req 0->159743) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.389990:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.390005:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.390014:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390021:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.390033:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.390038:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.390051:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-pcc.sh: write [0, 159631) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.390058:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390075:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390080:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000077e9c199[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390082:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000052eb6d97 17ffffc0001001 3:0 ffff9d8187722620 0 lru 00000080:00200000:2.0:1775764923.390086:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005f7d070a 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390088:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000077e9c199 00000080:00200000:2.0:1775764923.390089:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.390092:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390105:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390107:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d7a5202e[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390109:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000060aa63e5 17ffffc0001001 3:0 ffff9d8187722460 1 lru 00000080:00200000:2.0:1775764923.390113:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d3095260 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390114:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d7a5202e 00000080:00200000:2.0:1775764923.390115:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.390116:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390133:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390136:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008f2ba3af[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390138:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e03726ee 17ffffc0001001 3:0 ffff9d81877220e0 2 lru 00000080:00200000:2.0:1775764923.390141:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006f231d8e 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390143:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008f2ba3af 00000080:00200000:2.0:1775764923.390143:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.390144:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390152:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390154:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004958e1e8[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390155:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000048a3d661 17ffffc0001001 3:0 ffff9d8187722b60 3 lru 00000080:00200000:2.0:1775764923.390159:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f4eea801 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390160:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004958e1e8 00000080:00200000:2.0:1775764923.390160:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.390162:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390168:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390180:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000011591b[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390182:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000de1eb82b 17ffffc0001001 3:0 ffff9d81877229a0 4 lru 00000080:00200000:2.0:1775764923.390185:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009411110c 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390187:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000011591b 00000080:00200000:2.0:1775764923.390187:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.390189:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390197:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390199:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b6648e5a[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390200:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000dcf5906 17ffffc0001001 3:0 ffff9d8187722540 5 lru 00000080:00200000:2.0:1775764923.390203:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bb99244c 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390205:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b6648e5a 00000080:00200000:2.0:1775764923.390205:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.390206:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390213:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390215:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000010c62df8[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390216:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003d889582 17ffffc0001001 3:0 ffff9d8187722000 6 lru 00000080:00200000:2.0:1775764923.390220:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b419e2af 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390221:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000010c62df8 00000080:00200000:2.0:1775764923.390221:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.390222:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390229:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390231:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009214a133[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390232:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000571e0f71 17ffffc0001001 3:0 ffff9d81877222a0 7 lru 00000080:00200000:2.0:1775764923.390235:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008f9e32df 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390236:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009214a133 00000080:00200000:2.0:1775764923.390237:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.390238:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390244:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390246:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000441183c[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390248:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000051d0fa74 17ffffc0001001 3:0 ffff9d8187722ee0 8 lru 00000080:00200000:2.0:1775764923.390251:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007a3ae830 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390252:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000441183c 00000080:00200000:2.0:1775764923.390253:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.390254:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390265:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390267:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000002eae525[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390269:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000006f0fbac 17ffffc0001021 2:0 ffff9d8187722d20 9 lru 00000080:00200000:2.0:1775764923.390272:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000395a26f5 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390278:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000002eae525 00000080:00200000:2.0:1775764923.390279:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.390280:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390288:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390290:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000096432dfa[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390291:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000074d672ee 17ffffc0001001 3:0 ffff9d81877227e0 10 lru 00000080:00200000:2.0:1775764923.390295:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000003932ec 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390296:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000096432dfa 00000080:00200000:2.0:1775764923.390296:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.390297:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390309:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390311:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b984404c[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390312:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005610ea1a 17ffffc0001001 3:0 ffff9d81904f42a0 11 lru 00000080:00200000:2.0:1775764923.390315:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ec61bece 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390317:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b984404c 00000080:00200000:2.0:1775764923.390317:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.390318:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390330:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390333:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d10b689d[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390334:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005f09539c 17ffffc0001001 3:0 ffff9d81904f4ee0 12 lru 00000080:00200000:2.0:1775764923.390337:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ccf1153a 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390339:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d10b689d 00000080:00200000:2.0:1775764923.390339:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.390340:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390348:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390350:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000089458cf[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390352:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e034fbee 17ffffc0001001 3:0 ffff9d81904f4d20 13 lru 00000080:00200000:2.0:1775764923.390355:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f18f3ff3 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390356:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000089458cf 00000080:00200000:2.0:1775764923.390357:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.390358:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390364:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390366:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001039b08b[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390367:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008a320393 17ffffc0001001 3:0 ffff9d81904f47e0 14 lru 00000080:00200000:2.0:1775764923.390370:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000faa0397d 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390371:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001039b08b 00000080:00200000:2.0:1775764923.390372:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.390373:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390379:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390381:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000006f28b71[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390383:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000183cd59a 17ffffc0001001 3:0 ffff9d81904f48c0 15 lru 00000080:00200000:2.0:1775764923.390385:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000050972ca3 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390386:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000006f28b71 00000080:00200000:2.0:1775764923.390387:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.390388:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390395:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390397:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009fa0b35c[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390398:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000003e38bf7 17ffffc0001001 3:0 ffff9d81904f4c40 16 lru 00000080:00200000:2.0:1775764923.390401:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000020ed80ff 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390402:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009fa0b35c 00000080:00200000:2.0:1775764923.390403:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.390404:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390410:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390412:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bf6c95ad[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390414:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ebf786e6 17ffffc0001001 3:0 ffff9d81904f4a80 17 lru 00000080:00200000:2.0:1775764923.390416:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d5c02cda 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390418:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bf6c95ad 00000080:00200000:2.0:1775764923.390419:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.390420:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390426:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390428:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003a0c62d0[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390430:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b4b48fc2 17ffffc0001001 3:0 ffff9d81904f4380 18 lru 00000080:00200000:2.0:1775764923.390433:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000977630de 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390434:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003a0c62d0 00000080:00200000:2.0:1775764923.390435:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.390436:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390443:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390444:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007ab166c2[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390446:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000077689019 17ffffc0001001 3:0 ffff9d81904f4e00 19 lru 00000080:00200000:2.0:1775764923.390449:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001f2d18a1 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390450:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007ab166c2 00000080:00200000:2.0:1775764923.390451:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.390452:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390458:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390460:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000087f6fa0f[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390462:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c403725b 17ffffc0001001 3:0 ffff9d81904f41c0 20 lru 00000080:00200000:2.0:1775764923.390465:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fe033e50 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390466:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000087f6fa0f 00000080:00200000:2.0:1775764923.390467:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.390468:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390475:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390477:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dc7df806[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390478:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001cf758bb 17ffffc0001001 3:0 ffff9d81904f4700 21 lru 00000080:00200000:2.0:1775764923.390481:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000419b099a 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390483:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dc7df806 00000080:00200000:2.0:1775764923.390484:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.390485:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390493:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390495:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005cb347fc[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390497:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005fdcf8b1 17ffffc0001001 3:0 ffff9d81904f4620 22 lru 00000080:00200000:2.0:1775764923.390501:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000eb7f892 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390502:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005cb347fc 00000080:00200000:2.0:1775764923.390503:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.390504:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390513:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390516:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000096e479e8[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390518:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004421c891 17ffffc0001001 3:0 ffff9d81904f4460 23 lru 00000080:00200000:2.0:1775764923.390521:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000011c06671 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390522:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000096e479e8 00000080:00200000:2.0:1775764923.390523:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.390524:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390534:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390536:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000300c3024[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390538:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008c29a3ae 17ffffc0001021 2:0 ffff9d81904f40e0 24 lru 00000080:00200000:2.0:1775764923.390541:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fa977624 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390542:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000300c3024 00000080:00200000:2.0:1775764923.390543:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.390544:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390551:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390554:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001d35f3c6[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390556:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cf3e04fa 17ffffc0001001 3:0 ffff9d81904f4b60 25 lru 00000080:00200000:2.0:1775764923.390559:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003cc1a413 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390560:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001d35f3c6 00000080:00200000:2.0:1775764923.390561:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.390562:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390570:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390572:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007c140a35[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390574:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004fd3b2cf 17ffffc0001001 3:0 ffff9d81904f49a0 26 lru 00000100:00100000:1.0:1775764923.390577:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066210304:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.390580:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000afcd8721 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390584:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007c140a35 00000080:00200000:2.0:1775764923.390586:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.390587:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390595:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390599:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006e24ba69[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390602:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000072aec183 17ffffc0001001 3:0 ffff9d81904f4540 27 lru 00000080:00200000:2.0:1775764923.390608:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000093a0b782 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390609:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006e24ba69 00000080:00200000:2.0:1775764923.390610:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.390612:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390626:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390630:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a51fa51e[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390633:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e1826239 17ffffc0001001 3:0 ffff9d81904f4000 28 lru 00000080:00200000:2.0:1775764923.390638:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002969564e 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390640:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a51fa51e 00000080:00200000:2.0:1775764923.390641:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.390643:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390663:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390667:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001abdeb39[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390670:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d66b6e21 17ffffc0001001 3:0 ffff9d81887197e0 29 lru 00000080:00200000:2.0:1775764923.390675:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002260af02 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390677:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001abdeb39 00000080:00200000:2.0:1775764923.390678:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.390680:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390693:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390696:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004e709906[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390698:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005375f05a 17ffffc0001001 3:0 ffff9d81887198c0 30 lru 00000080:00200000:2.0:1775764923.390701:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004ad0d888 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390702:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004e709906 00000080:00200000:2.0:1775764923.390703:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.390704:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390711:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390713:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004b291a4a[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390715:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000008f63655 17ffffc0001001 3:0 ffff9d8188719c40 31 lru 00000080:00200000:2.0:1775764923.390718:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000586cd172 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390719:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004b291a4a 00000080:00200000:2.0:1775764923.390719:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.390720:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390727:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390729:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000996b71af[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390730:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000997f1b08 17ffffc0001001 3:0 ffff9d8188719a80 32 lru 00000080:00200000:2.0:1775764923.390733:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d10f8427 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390734:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000996b71af 00000080:00200000:2.0:1775764923.390735:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.390736:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390742:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390744:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000017d2ffe8[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390745:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001be641f0 17ffffc0001001 3:0 ffff9d8188719380 33 lru 00000080:00200000:2.0:1775764923.390748:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e8c11183 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390750:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000017d2ffe8 00000080:00200000:2.0:1775764923.390751:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.390751:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390759:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390763:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fd73f39e[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390765:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d0b57282 17ffffc0001001 3:0 ffff9d8188719e00 34 lru 00000080:00200000:2.0:1775764923.390770:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000045c6de3c 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390772:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fd73f39e 00000080:00200000:2.0:1775764923.390773:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.390775:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390788:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390792:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000469f539[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390795:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009adb06ac 17ffffc0001001 3:0 ffff9d81887191c0 35 lru 00000080:00200000:2.0:1775764923.390801:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006cdec9b8 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390803:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000469f539 00000080:00200000:2.0:1775764923.390805:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.390806:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390829:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390834:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000446f4044[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390837:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000010386051 17ffffc0001001 3:0 ffff9d8188719700 36 lru 00000080:00200000:2.0:1775764923.390842:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005b163699 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390844:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000446f4044 00000080:00200000:2.0:1775764923.390845:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.390847:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.390860:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.390863:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009c1e7b3b[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390864:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000089b5adde 17ffffc0001001 3:0 ffff9d8188719620 37 lru 00000080:00200000:2.0:1775764923.390868:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000c4e5938 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390869:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009c1e7b3b 00000080:00200000:2.0:1775764923.390870:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.390871:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 3983 bytes 00000080:00200000:2.0:1775764923.390897:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 3983, copied 3983 00000080:00200000:2.0:1775764923.390900:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dd3d1308[3 00000000efccfc63 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.390902:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f0bf58f5 17ffffc0001001 3:0 ffff9d8188719460 38 lru 00000080:00200000:2.0:1775764923.390905:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000544257ef 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 000000009452db5f 000000007bcb0a35 > 3< 0 0 > 4< 0 0 8 2256896 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.390907:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dd3d1308 00000080:00200000:2.0:1775764923.390908:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.390911:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 39, from 0, to 3983 prio 0 00000008:00010000:2.0:1775764923.390921:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390930:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390935:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390940:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390945:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390949:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390953:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390957:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390961:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390965:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390969:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390973:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390978:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390982:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.390986:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.390993:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f2bb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.390997:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391001:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391005:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391010:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391014:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391022:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391026:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391030:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391034:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391038:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391042:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391046:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391050:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391054:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391059:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.391066:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f2bb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.391068:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391072:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391076:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391080:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391084:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391088:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391092:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391099:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.391103:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000000a6f1be4 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.391108:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f2bb8, count 9, dirtied 9 00000080:00200000:2.0:1775764923.391110:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 39 pages 159631 bytes, tot: 159631 00000080:00200000:2.0:1775764923.391112:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-pcc.sh: write bytes 0, result: 159631 00000020:00010000:2.0:1775764923.391117:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.391119:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.391121:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cec30 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.391123:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.391128:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.391133:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.391135:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cec30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.391139:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 159631 bytes 00000080:00200000:2.0:1775764923.391141:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 38] 00000080:00200000:2.0:1775764923.391151:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x42:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.391154:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-pcc.sh: 2 io complete with rc: 0, result: 159631, restart: 0 00000080:00200000:2.0:1775764923.391158:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x42:0x0] updating i_size 159631 i_blocks 0 00000080:00200000:2.0:1775764923.391160:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 159631 00000080:00200008:2.0:1775764923.391189:0:34105:0:(file.c:447:ll_file_release()) START file sanity-pcc.sh:[0x240004281:0x42:0x0](ffff9d81843f2a08), flags 100001 00000002:00100000:2.0:1775764923.391207:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906f8380 x1862023066209920/t38654705889(38654705889) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.391213:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.391216:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906f9180 x1862023066210560/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.391225:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906f9180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066210560:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.391256:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.391941:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906f9180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066210560:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.391951:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906f8380 x1862023066209920/t38654705889(38654705889) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.391974:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.391980:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.391986:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.391992:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.391996:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.391999:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.392000:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.392003:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.392005:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.392013:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.392018:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.392021:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.392023:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.392025:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.392026:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.392028:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-pcc.sh, flags=8 00000080:00200008:2.0:1775764923.392031:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-pcc.sh:[0x240004281:0x42:0x0](ffff9d81843f2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.392034:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x42:0x0](ffff9d81843f2a08),name=sanity-pcc.sh 00000002:00010000:2.0:1775764923.392040:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x42:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.392046:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.392053:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.392059:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.392064:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x42:0x0](ffff9d81843f2a08) 00000080:00010000:2.0:1775764923.392067:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x42:0x0](ffff9d81843f2a08) for lock 0xeef44110eff5638 00000080:00010000:2.0:1775764923.392070:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5638 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.392073:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.392081:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.392102:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x42:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.392106:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x42:0x0] 00000008:00010000:2.0:1775764923.392116:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.392117:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cec30, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.392124:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.392130:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.392135:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.392137:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.392142:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2aef4e0/0xeef44110eff5646 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->159743) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321558dd expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.392147:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.392149:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x42:0x0] updating i_size 159631 i_blocks 0 00000020:00010000:2.0:1775764923.392151:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.392153:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.392155:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cec30 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.392156:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.392158:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cec30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.392183:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x42:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.392220:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.392229:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.392236:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.392244:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.392250:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.392255:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.392257:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.392259:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.392261:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.392266:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.392271:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.392273:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.392276:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.392277:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.392278:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.392284:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-pfl.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.392286:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-pfl.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.392292:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-pfl.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.392307:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.392312:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.392317:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.392328:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066210688:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.392365:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.393121:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066210688:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.393131:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393141:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393147:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393154:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393159:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393164:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583ba00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.393167:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393184:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393190:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393197:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.393200:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393206:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ba00/0xeef44110eff564d lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.393218:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fbb80 x1862023066210688/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.393227:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.393237:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.393260:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.393267:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.393274:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.393283:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.393289:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.393293:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.393296:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.393299:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.393302:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.393310:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.393319:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.393323:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.393326:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.393329:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.393331:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.393337:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-pfl.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.393340:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-pfl.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.393346:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-pfl.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.393362:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393368:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.393373:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.393381:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066210816:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.393409:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.394199:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066210816:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.394205:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394210:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394214:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394220:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394226:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394231:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583b600 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.394234:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394239:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394244:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394250:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.394253:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394259:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff5654 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.394270:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fbb80 x1862023066210816/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.394278:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.394288:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.394341:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.394349:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.394356:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.394365:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.394371:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.394376:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.394378:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.394382:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.394385:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.394393:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.394401:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.394404:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.394408:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.394410:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.394413:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.394419:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.394423:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-pfl.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da000, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.394427:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-pfl.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.394436:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-pfl.sh,[0x240004281:0x43:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.394452:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.394456:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.394461:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.394466:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fbb80 x1862023066211072/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.394474:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.394493:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211072:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.394560:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.395566:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fbb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211072:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.395574:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395587:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 4/0,1 mode: --/PR res: [0x240004281:0x43:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395592:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 4/0,1 mode: --/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395598:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395602:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395608:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395616:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395620:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.395624:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fbb80 x1862023066211072/t38654705891(38654705891) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.395630:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fbb80 x1862023066211072/t38654705891(38654705891) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.395636:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.395642:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395646:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/67 (0 0) 00000080:00200000:2.0:1775764923.395649:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.395653:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-pfl.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.395664:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x43:0x0](ffff9d81843f5348) 00000080:00200000:2.0:1775764923.395721:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x43:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.395728:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x43:0x0](ffff9d81843f5348) 00000080:00010000:2.0:1775764923.395730:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x43:0x0](ffff9d81843f5348) for lock 0xeef44110eff565b 00000080:00010000:2.0:1775764923.395734:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x43:0x0](ffff9d81843f5348) 00000080:00200008:2.0:1775764923.395739:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-pfl.sh:[0x240004281:0x43:0x0](ffff9d81843f5348), flags 100301 00000002:00100000:2.0:1775764923.395747:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fbb80 x1862023066211072/t38654705891(38654705891) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.395766:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff565b from it ffff9d8190e6e240 00010000:00010000:2.0:1775764923.395768:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395773:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.395785:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x43:0x0](ffff9d81843f5348), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.395794:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-pfl.sh:[0x240004281:0x43:0x0](ffff9d81843f5348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.395798:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x43:0x0](ffff9d81843f5348),name=sanity-pfl.sh 00000002:00010000:2.0:1775764923.395806:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x43:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.395812:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.395820:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.395827:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.395831:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x43:0x0](ffff9d81843f5348) 00000080:00010000:2.0:1775764923.395834:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x43:0x0](ffff9d81843f5348) for lock 0xeef44110eff565b 00000080:00010000:2.0:1775764923.395837:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff565b from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.395841:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.395847:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.395868:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x43:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.395891:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x43:0x0] 00000008:00010000:2.0:1775764923.395907:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.395910:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cefb0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.395931:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.395939:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.395957:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1775764923.396655:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.396664:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.396672:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.396681:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.396688:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.396693:0:10947:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.396700:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.396705:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.396709:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.396713:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.396721:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066211200:192.168.202.105@tcp:101:cp.0 00000080:00200000:2.0:1775764923.396725:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x43:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.396728:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.396731:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.396733:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cefb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.396736:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.396738:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cefb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.396748:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x43:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.396795:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-pfl.sh:[0x240004281:0x43:0x0], ppos: 0, count: 84654 00000080:00200000:2.0:1775764923.396808:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-pfl.sh: write ppos: 0, bytes: 84654 00000080:00200000:2.0:1775764923.396810:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-pfl.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.396814:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@0000000086437d3b[0x110, 1, [0x240004281:0x43:0x0] hash] 00000020:00200000:2.0:1775764923.396816:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 84654) 0 0 00000080:00200000:2.0:1775764923.396818:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x43:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.396825:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 20] 00020000:00200000:2.0:1775764923.396829:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 84654) 84654, 84654 00020000:00200000:2.0:1775764923.396833:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.396848:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 20] 00000008:00010000:2.0:1775764923.396855:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.396858:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cefb0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.396865:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/546 (0 86015) 00010000:00010000:2.0:1775764923.396913:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.396921:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->86015] (req 0->86015) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.396929:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->86015] (req 0->86015) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.396942:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211328:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.396972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1f00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.397621:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.397629:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082328576 00000100:00100000:1.0:1775764923.397631:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082328576 00000100:00100000:1.0:1775764923.397654:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 287 00000100:00100000:1.0:1775764923.397659:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082328576:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.397666:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.397693:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.397697:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796dc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082328576:12345-192.168.202.105@tcp:104: Request processed in 39us (121us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.397701:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 287 00010000:00010000:2.0:1775764923.397719:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.397726:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.397735:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583aa00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.397740:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.397769:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x43:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.397774:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x43:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.397814:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.397820:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.397835:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.397840:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff5662 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155931 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.397850:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.397853:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.397854:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.398245:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211328:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.398258:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->86015] (req 0->86015) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.398269:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->86015] (req 0->86015) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.398277:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->86015] (req 0->86015) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.398683:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.398699:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.398710:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.398721:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.398727:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.398733:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.398750:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-pfl.sh: write [0, 84654) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.398759:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.398793:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.398800:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009ddcfdbb[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.398804:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000017aa50f4 17ffffc0001021 2:0 ffff9d81887190e0 0 lru 00000080:00200000:2.0:1775764923.398811:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004c095f77 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.398813:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009ddcfdbb 00000080:00200000:2.0:1775764923.398814:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.398816:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.398836:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.398840:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c4fda280[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.398843:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003355afe6 17ffffc0001001 3:0 ffff9d8188719b60 1 lru 00000080:00200000:2.0:1775764923.398848:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fdc19ab0 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.398850:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c4fda280 00000080:00200000:2.0:1775764923.398851:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.398853:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.398862:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.398866:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000090d31fb[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.398868:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007d25182b 17ffffc0001001 3:0 ffff9d81887199a0 2 lru 00000080:00200000:2.0:1775764923.398889:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006971dee6 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.398892:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000090d31fb 00000080:00200000:2.0:1775764923.398893:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.398895:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.398908:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.398911:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ced44c28[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.398912:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a1660b92 17ffffc0001001 3:0 ffff9d8188719540 3 lru 00000080:00200000:2.0:1775764923.398917:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ceef8e3d 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.398919:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ced44c28 00000080:00200000:2.0:1775764923.398920:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.398922:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.398934:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.398938:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ffbe1fdd[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.398941:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006753b059 17ffffc0001001 3:0 ffff9d8188719000 4 lru 00000080:00200000:2.0:1775764923.398946:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000838bfb2 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.398948:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ffbe1fdd 00000080:00200000:2.0:1775764923.398949:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.398951:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.398966:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.398970:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b95e911a[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.398973:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003260e3d6 17ffffc0001001 3:0 ffff9d81887192a0 5 lru 00000080:00200000:2.0:1775764923.398978:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f321213c 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.398980:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b95e911a 00000080:00200000:2.0:1775764923.398981:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.398983:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.398996:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399000:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008b3ff2b5[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399002:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000047e7b534 17ffffc0001001 3:0 ffff9d8188719ee0 6 lru 00000080:00200000:2.0:1775764923.399008:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000671ee7f2 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399010:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008b3ff2b5 00000080:00200000:2.0:1775764923.399011:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.399013:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399022:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399025:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000039d4dfe7[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399028:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000034141dd0 17ffffc0001001 3:0 ffff9d8188719d20 7 lru 00000080:00200000:2.0:1775764923.399033:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002dc41c17 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399036:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000039d4dfe7 00000080:00200000:2.0:1775764923.399037:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.399038:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399061:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399065:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000347f79eb[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399069:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000423c9b04 17ffffc0001001 3:0 ffff9d8190734540 8 lru 00000080:00200000:2.0:1775764923.399074:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cf2a6563 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399076:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000347f79eb 00000080:00200000:2.0:1775764923.399077:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.399079:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399089:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399093:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006528c437[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399101:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000022ba66e1 17ffffc0001001 3:0 ffff9d8190734000 9 lru 00000080:00200000:2.0:1775764923.399109:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d977f6fe 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399111:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006528c437 00000080:00200000:2.0:1775764923.399113:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.399115:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399127:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399131:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003d8707b8[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399134:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a41f2c20 17ffffc0001001 3:0 ffff9d81907342a0 10 lru 00000080:00200000:2.0:1775764923.399139:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fcbd6392 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399142:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003d8707b8 00000080:00200000:2.0:1775764923.399143:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.399145:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000100:00100000:1.0:1775764923.399147:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ffb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066211456:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.399155:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399160:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002c208367[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399162:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000061bd2d8f 17ffffc0001001 3:0 ffff9d8190734ee0 11 lru 00000080:00200000:2.0:1775764923.399166:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000035c4465b 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399168:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002c208367 00000080:00200000:2.0:1775764923.399179:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.399180:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399191:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399194:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004ead24db[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399195:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000262c855a 17ffffc0001001 3:0 ffff9d8190734d20 12 lru 00000080:00200000:2.0:1775764923.399199:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000042f992d1 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399200:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004ead24db 00000080:00200000:2.0:1775764923.399201:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.399202:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399209:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399210:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009172a515[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399212:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000618704e3 17ffffc0001001 3:0 ffff9d81907347e0 13 lru 00000080:00200000:2.0:1775764923.399215:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004394caf8 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399217:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009172a515 00000080:00200000:2.0:1775764923.399217:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.399218:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399225:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399226:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000601a654f[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399228:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007ca56906 17ffffc0001001 3:0 ffff9d81907348c0 14 lru 00000080:00200000:2.0:1775764923.399231:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a1e204f6 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399232:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000601a654f 00000080:00200000:2.0:1775764923.399232:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.399233:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399242:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399244:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000097ab3af6[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399246:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004fd9151e 17ffffc0001021 2:0 ffff9d8190734c40 15 lru 00000080:00200000:2.0:1775764923.399249:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006ae6f029 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399250:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000097ab3af6 00000080:00200000:2.0:1775764923.399250:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.399252:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399258:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399260:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fcfc541d[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399262:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000033c98cd7 17ffffc0001001 3:0 ffff9d8190734a80 16 lru 00000080:00200000:2.0:1775764923.399265:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e98d6a25 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399266:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fcfc541d 00000080:00200000:2.0:1775764923.399267:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.399268:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399274:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399276:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ec39f1f6[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399277:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002d3d03a4 17ffffc0001001 3:0 ffff9d8190734380 17 lru 00000080:00200000:2.0:1775764923.399280:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000055c1cf89 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399282:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ec39f1f6 00000080:00200000:2.0:1775764923.399282:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.399283:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399290:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399292:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f68c780d[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399293:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000025d4ce7a 17ffffc0001001 3:0 ffff9d8190734e00 18 lru 00000080:00200000:2.0:1775764923.399296:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c043bb9b 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399297:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f68c780d 00000080:00200000:2.0:1775764923.399298:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.399299:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.399306:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.399308:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000741ea63d[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399309:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f45a18b7 17ffffc0001001 3:0 ffff9d81907341c0 19 lru 00000080:00200000:2.0:1775764923.399312:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f434f79f 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399313:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000741ea63d 00000080:00200000:2.0:1775764923.399313:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.399314:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 2734 bytes 00000080:00200000:2.0:1775764923.399323:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 2734, copied 2734 00000080:00200000:2.0:1775764923.399325:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004e4fcd41[3 0000000024693d14 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.399326:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005e75c334 17ffffc0001001 3:0 ffff9d8190734700 20 lru 00000080:00200000:2.0:1775764923.399330:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000042d29b04 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 00000000409b8422 00000000527eb06f > 3< 0 0 > 4< 0 0 8 3153920 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.399331:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004e4fcd41 00000080:00200000:2.0:1775764923.399332:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.399334:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 21, from 0, to 2734 prio 0 00000008:00010000:2.0:1775764923.399343:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399351:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399357:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399361:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399365:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399369:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399373:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399377:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399390:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399397:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399405:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399414:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399422:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399431:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399439:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.399447:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f54f8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.399451:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399459:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399464:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399468:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399472:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.399476:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000614d3869 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.399480:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f54f8, count 6, dirtied 6 00000080:00200000:2.0:1775764923.399482:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 21 pages 84654 bytes, tot: 84654 00000080:00200000:2.0:1775764923.399484:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-pfl.sh: write bytes 0, result: 84654 00000020:00010000:2.0:1775764923.399490:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.399491:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.399493:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cefb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.399495:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.399500:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.399509:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.399511:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cefb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.399514:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 84654 bytes 00000080:00200000:2.0:1775764923.399517:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 20] 00000080:00200000:2.0:1775764923.399526:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x43:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.399528:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-pfl.sh: 2 io complete with rc: 0, result: 84654, restart: 0 00000080:00200000:2.0:1775764923.399532:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x43:0x0] updating i_size 84654 i_blocks 0 00000080:00200000:2.0:1775764923.399534:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 84654 00000080:00200008:2.0:1775764923.399548:0:34105:0:(file.c:447:ll_file_release()) START file sanity-pfl.sh:[0x240004281:0x43:0x0](ffff9d81843f5348), flags 100001 00000002:00100000:2.0:1775764923.399565:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fbb80 x1862023066211072/t38654705891(38654705891) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.399571:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.399574:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fc700 x1862023066211712/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.399583:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fc700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211712:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.399612:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.400300:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fc700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211712:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.400315:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fbb80 x1862023066211072/t38654705891(38654705891) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.400347:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.400356:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.400365:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.400381:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.400388:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.400392:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.400395:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.400399:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.400402:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.400410:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.400418:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.400422:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.400426:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.400429:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.400432:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.400435:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-pfl.sh, flags=8 00000080:00200008:2.0:1775764923.400439:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-pfl.sh:[0x240004281:0x43:0x0](ffff9d81843f5348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.400443:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x43:0x0](ffff9d81843f5348),name=sanity-pfl.sh 00000002:00010000:2.0:1775764923.400451:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x43:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.400456:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.400464:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.400471:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.400474:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x43:0x0](ffff9d81843f5348) 00000080:00010000:2.0:1775764923.400475:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x43:0x0](ffff9d81843f5348) for lock 0xeef44110eff565b 00000080:00010000:2.0:1775764923.400478:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff565b from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.400479:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.400490:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.400506:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x43:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.400510:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x43:0x0] 00000008:00010000:2.0:1775764923.400518:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.400520:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cefb0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.400526:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.400532:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.400537:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.400539:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.400544:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000a87325b3/0xeef44110eff5669 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->86015) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155938 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.400552:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.400555:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x43:0x0] updating i_size 84654 i_blocks 0 00000020:00010000:2.0:1775764923.400557:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.400559:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.400561:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cefb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.400563:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.400564:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cefb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.400572:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x43:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.400597:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.400602:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.400606:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.400611:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.400615:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.400617:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.400619:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.400621:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.400623:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.400627:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.400634:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.400636:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.400638:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.400640:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.400641:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.400646:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-quota.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.400648:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-quota.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.400653:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-quota.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.400667:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.400671:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.400676:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.400683:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fe300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211840:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.400716:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.401538:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fe300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211840:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.401545:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401550:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401554:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401560:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401566:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401572:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583b200 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.401575:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401580:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401586:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401592:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.401596:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401602:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff5670 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.401613:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fe300 x1862023066211840/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.401622:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.401632:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.401652:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.401658:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.401664:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.401673:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.401679:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.401683:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.401684:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.401687:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.401689:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.401693:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.401698:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.401701:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.401703:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.401704:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.401706:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.401710:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-quota.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.401712:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-quota.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.401716:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-quota.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.401727:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401731:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.401738:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.401745:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fe300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211968:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.401772:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.402587:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fe300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066211968:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.402593:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402598:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402602:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402605:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402608:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402611:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583be00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.402613:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402618:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402623:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402629:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.402632:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402637:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff5677 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.402648:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fe300 x1862023066211968/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.402657:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.402666:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.402700:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.402707:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.402712:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.402717:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.402721:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.402724:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.402726:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.402728:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.402730:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.402734:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.402739:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.402742:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.402744:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.402745:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.402747:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.402752:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.402755:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-quota.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851dac00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.402760:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-quota.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.402771:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-quota.sh,[0x240004281:0x44:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.402792:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.402799:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.402806:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.402812:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fe300 x1862023066212224/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.402820:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.402831:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fe300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066212224:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.402905:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.403873:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fe300 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066212224:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.403893:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.403905:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 4/0,1 mode: --/PR res: [0x240004281:0x44:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.403910:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 4/0,1 mode: --/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.403915:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.403920:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.403925:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.403928:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.403933:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.403937:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fe300 x1862023066212224/t38654705893(38654705893) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.403944:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fe300 x1862023066212224/t38654705893(38654705893) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.403950:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.403956:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.403960:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/68 (0 0) 00000080:00200000:2.0:1775764923.403963:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.403967:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-quota.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.403981:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x44:0x0](ffff9d81843f5b88) 00000080:00200000:2.0:1775764923.404046:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x44:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.404056:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x44:0x0](ffff9d81843f5b88) 00000080:00010000:2.0:1775764923.404059:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x44:0x0](ffff9d81843f5b88) for lock 0xeef44110eff567e 00000080:00010000:2.0:1775764923.404070:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x44:0x0](ffff9d81843f5b88) 00000080:00200008:2.0:1775764923.404077:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-quota.sh:[0x240004281:0x44:0x0](ffff9d81843f5b88), flags 100301 00000002:00100000:2.0:1775764923.404089:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fe300 x1862023066212224/t38654705893(38654705893) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.404116:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff567e from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.404120:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.404128:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.404141:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x44:0x0](ffff9d81843f5b88), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.404150:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-quota.sh:[0x240004281:0x44:0x0](ffff9d81843f5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.404155:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x44:0x0](ffff9d81843f5b88),name=sanity-quota.sh 00000002:00010000:2.0:1775764923.404162:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x44:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.404167:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.404189:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.404196:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.404199:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x44:0x0](ffff9d81843f5b88) 00000080:00010000:2.0:1775764923.404202:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x44:0x0](ffff9d81843f5b88) for lock 0xeef44110eff567e 00000080:00010000:2.0:1775764923.404206:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff567e from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.404209:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.404215:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.404236:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x44:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.404242:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x44:0x0] 00000008:00010000:2.0:1775764923.404256:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.404258:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ceeb0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.404280:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.404289:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.404317:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.404337:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066212352:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.405021:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.405040:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.405055:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.405064:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.405076:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.405091:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.405102:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.405111:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.405119:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.405123:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x44:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.405130:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00010000:00010000:1.0:1775764923.405131:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.405136:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.405143:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ceeb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.405149:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000100:00100000:1.0:1775764923.405152:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066212352:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.405154:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ceeb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.405191:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x44:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.405299:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-quota.sh:[0x240004281:0x44:0x0], ppos: 0, count: 244888 00000080:00200000:2.0:1775764923.405314:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-quota.sh: write ppos: 0, bytes: 244888 00000080:00200000:2.0:1775764923.405317:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-quota.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.405322:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@000000002d79c8e7[0x110, 1, [0x240004281:0x44:0x0] hash] 00000020:00200000:2.0:1775764923.405324:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 244888) 0 0 00000080:00200000:2.0:1775764923.405327:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x44:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.405333:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 59] 00020000:00200000:2.0:1775764923.405336:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 244888) 244888, 244888 00020000:00200000:2.0:1775764923.405340:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.405352:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 59] 00000008:00010000:2.0:1775764923.405356:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.405359:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ceeb0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.405364:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/547 (0 245759) 00010000:00010000:2.0:1775764923.405377:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.405382:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->245759] (req 0->245759) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.405387:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 3/0,1 mode: --/PW res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->245759] (req 0->245759) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.405395:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fc380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066212480:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.405420:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.406047:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082328832 00000100:00100000:1.0:1775764923.406051:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082328832 00000100:00100000:1.0:1775764923.406076:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 289 00000100:00100000:1.0:1775764923.406084:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082328832:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.406093:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.406137:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.406144:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082328832:12345-192.168.202.105@tcp:104: Request processed in 60us (133us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.406151:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 289 00010000:00010000:2.0:1775764923.406160:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.406165:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.406184:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583a000 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.406188:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.406215:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.406221:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.406255:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.406263:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.406284:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.406291:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff5685 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa3215598c expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.406301:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.406302:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.406305:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066212608:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.406306:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.406854:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fc380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066212480:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.406862:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->245759] (req 0->245759) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.406871:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->245759] (req 0->245759) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.406901:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: --/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->245759] (req 0->245759) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.407224:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.407231:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.407238:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.407253:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.407266:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.407271:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.407287:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-quota.sh: write [0, 244888) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.407296:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407321:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407329:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007a6c7a6f[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407332:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000372e177c 17ffffc0001001 3:0 ffff9d8190734620 0 lru 00000080:00200000:2.0:1775764923.407338:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000041713d99 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407340:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007a6c7a6f 00000080:00200000:2.0:1775764923.407342:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.407344:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407363:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407367:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001f5817c0[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407370:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a45761df 17ffffc0001001 3:0 ffff9d8190734460 1 lru 00000080:00200000:2.0:1775764923.407375:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008397a7b7 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407377:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001f5817c0 00000080:00200000:2.0:1775764923.407378:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.407379:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407389:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407391:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a0b6a0e2[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407393:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e9ca1c17 17ffffc0001001 3:0 ffff9d81907340e0 2 lru 00000080:00200000:2.0:1775764923.407396:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004e713843 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407397:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a0b6a0e2 00000080:00200000:2.0:1775764923.407398:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.407399:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407405:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407407:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000089a68b0[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407408:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000053c5713f 17ffffc0001001 3:0 ffff9d8190734b60 3 lru 00000080:00200000:2.0:1775764923.407411:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000236207c1 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407412:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000089a68b0 00000080:00200000:2.0:1775764923.407413:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.407414:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407420:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407422:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000398b2526[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407424:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000000444315 17ffffc0001001 3:0 ffff9d81907349a0 4 lru 00000080:00200000:2.0:1775764923.407427:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005955ad25 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407429:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000398b2526 00000080:00200000:2.0:1775764923.407429:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.407430:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407445:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407447:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e830b5a5[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407448:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000af2d3843 17ffffc0001001 3:0 ffff9d8186c7aee0 5 lru 00000080:00200000:2.0:1775764923.407452:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cbc4b467 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407454:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e830b5a5 00000080:00200000:2.0:1775764923.407455:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.407457:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407470:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407473:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d4dd917d[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407476:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005da5aab7 17ffffc0001001 3:0 ffff9d8186c7ad20 6 lru 00000080:00200000:2.0:1775764923.407481:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002f2797e4 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407483:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d4dd917d 00000080:00200000:2.0:1775764923.407484:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.407486:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407499:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407503:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006aed6a16[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407506:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a826aef7 17ffffc0001001 3:0 ffff9d8186c7a7e0 7 lru 00000080:00200000:2.0:1775764923.407511:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000056dc7f57 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407513:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006aed6a16 00000080:00200000:2.0:1775764923.407514:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.407515:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407530:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407535:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e66df1dc[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407538:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007e4650bf 17ffffc0001001 3:0 ffff9d8186c7a8c0 8 lru 00000080:00200000:2.0:1775764923.407543:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002fb88498 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407544:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e66df1dc 00000080:00200000:2.0:1775764923.407545:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.407547:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407566:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407570:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006356f289[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407572:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e89b72b5 17ffffc0001021 2:0 ffff9d8186c7ac40 9 lru 00000080:00200000:2.0:1775764923.407576:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000037f23b3c 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407577:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006356f289 00000080:00200000:2.0:1775764923.407578:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.407579:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407588:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407590:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000060f2e0ca[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407593:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a541af29 17ffffc0001001 3:0 ffff9d8186c7aa80 10 lru 00000080:00200000:2.0:1775764923.407598:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f41c7e1a 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407600:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000060f2e0ca 00000080:00200000:2.0:1775764923.407601:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.407603:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407615:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407618:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000130f8453[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407621:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000059debe29 17ffffc0001001 3:0 ffff9d8186c7a380 11 lru 00000080:00200000:2.0:1775764923.407626:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000482efa66 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407628:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000130f8453 00000080:00200000:2.0:1775764923.407629:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.407631:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407645:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407649:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000041931152[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407651:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004f2e0dbc 17ffffc0001001 3:0 ffff9d8186c7ae00 12 lru 00000080:00200000:2.0:1775764923.407656:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c35cca6a 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407659:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000041931152 00000080:00200000:2.0:1775764923.407660:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.407661:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407672:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407675:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000223b7dba[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407677:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000dbdf5b10 17ffffc0001001 3:0 ffff9d8186c7a1c0 13 lru 00000080:00200000:2.0:1775764923.407680:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000023ecfdb4 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407682:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000223b7dba 00000080:00200000:2.0:1775764923.407683:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.407684:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407691:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407693:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a51c5310[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407694:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000012c40cb2 17ffffc0001001 3:0 ffff9d8186c7a700 14 lru 00000080:00200000:2.0:1775764923.407697:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000084e26bea 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407699:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a51c5310 00000080:00200000:2.0:1775764923.407699:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.407700:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407707:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407709:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000023f2fe02[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407710:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000da7d7299 17ffffc0001001 3:0 ffff9d8186c7a620 15 lru 00000080:00200000:2.0:1775764923.407713:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000062fb294f 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407715:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000023f2fe02 00000080:00200000:2.0:1775764923.407716:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.407717:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407724:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407727:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dc9d91bd[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407729:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001b6089d8 17ffffc0001001 3:0 ffff9d8186c7a460 16 lru 00000080:00200000:2.0:1775764923.407732:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c2ae8b45 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407733:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dc9d91bd 00000080:00200000:2.0:1775764923.407734:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.407735:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407743:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407746:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f417d9a1[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407747:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001d062cb2 17ffffc0001001 3:0 ffff9d8186c7a0e0 17 lru 00000080:00200000:2.0:1775764923.407750:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c0858f52 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407752:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f417d9a1 00000080:00200000:2.0:1775764923.407753:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.407755:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407763:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407765:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000000c063ec[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407767:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b6721cba 17ffffc0001001 3:0 ffff9d8186c7ab60 18 lru 00000080:00200000:2.0:1775764923.407770:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a7bf939e 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407772:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000000c063ec 00000080:00200000:2.0:1775764923.407773:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.407774:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407782:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000100:00100000:1.0:1775764923.407785:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066212608:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.407786:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000057b92540[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407789:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000133ba04b 17ffffc0001001 3:0 ffff9d8186c7a9a0 19 lru 00000080:00200000:2.0:1775764923.407795:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002c3335fc 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407797:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000057b92540 00000080:00200000:2.0:1775764923.407798:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.407799:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407807:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407809:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bd90aec8[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407811:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a287a61a 17ffffc0001001 3:0 ffff9d8186c7a540 20 lru 00000080:00200000:2.0:1775764923.407815:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004303d5e7 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407816:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bd90aec8 00000080:00200000:2.0:1775764923.407817:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.407818:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407824:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407826:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bd42811a[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407828:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005dbdb256 17ffffc0001001 3:0 ffff9d8186c7a000 21 lru 00000080:00200000:2.0:1775764923.407831:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001ff6e1c5 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407832:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bd42811a 00000080:00200000:2.0:1775764923.407833:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.407834:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407840:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407842:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ed0934f9[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407844:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006142ab4e 17ffffc0001001 3:0 ffff9d8186c7a2a0 22 lru 00000080:00200000:2.0:1775764923.407847:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004d91442a 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407848:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ed0934f9 00000080:00200000:2.0:1775764923.407849:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.407850:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407862:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407864:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b830eb52[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407865:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001e0f1e7f 17ffffc0001001 3:0 ffff9d818877d700 23 lru 00000080:00200000:2.0:1775764923.407868:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b85ccd26 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407870:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b830eb52 00000080:00200000:2.0:1775764923.407870:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.407871:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407899:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407902:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000040753043[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407903:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004ddc073c 17ffffc0001021 2:0 ffff9d818877d620 24 lru 00000080:00200000:2.0:1775764923.407907:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003149317d 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407908:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000040753043 00000080:00200000:2.0:1775764923.407909:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.407910:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407918:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407920:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004c738bb6[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407922:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006cae793e 17ffffc0001001 3:0 ffff9d818877d460 25 lru 00000080:00200000:2.0:1775764923.407925:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ea62d150 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407926:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004c738bb6 00000080:00200000:2.0:1775764923.407927:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.407928:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407935:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407937:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000070839955[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407938:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000084db765c 17ffffc0001001 3:0 ffff9d818877d0e0 26 lru 00000080:00200000:2.0:1775764923.407941:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e17bebb4 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407942:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000070839955 00000080:00200000:2.0:1775764923.407943:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.407945:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407951:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407953:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005bbcfb8f[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407955:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ddad378a 17ffffc0001001 3:0 ffff9d818877db60 27 lru 00000080:00200000:2.0:1775764923.407958:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ca37e3de 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407959:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005bbcfb8f 00000080:00200000:2.0:1775764923.407959:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.407960:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407966:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407968:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000390db086[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407970:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f2dd93f6 17ffffc0001001 3:0 ffff9d818877d9a0 28 lru 00000080:00200000:2.0:1775764923.407973:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000186de99d 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.407974:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000390db086 00000080:00200000:2.0:1775764923.407975:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.407976:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.407987:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.407990:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a5f86bd5[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.407993:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000123314d8 17ffffc0001001 3:0 ffff9d818877d540 29 lru 00000080:00200000:2.0:1775764923.407998:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c7b09b2a 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408001:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a5f86bd5 00000080:00200000:2.0:1775764923.408001:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.408003:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408016:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408020:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000398fe0ca[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408022:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b2982d88 17ffffc0001001 3:0 ffff9d818877d000 30 lru 00000080:00200000:2.0:1775764923.408026:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004c051446 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408028:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000398fe0ca 00000080:00200000:2.0:1775764923.408028:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.408030:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408039:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408042:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000646706cf[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408045:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000878ea2d9 17ffffc0001001 3:0 ffff9d818877d2a0 31 lru 00000080:00200000:2.0:1775764923.408050:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000b078e68 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408052:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000646706cf 00000080:00200000:2.0:1775764923.408053:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.408055:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408066:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408070:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c219cc78[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408073:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002802d64f 17ffffc0001001 3:0 ffff9d818877dee0 32 lru 00000080:00200000:2.0:1775764923.408078:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005bb23cea 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408080:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c219cc78 00000080:00200000:2.0:1775764923.408082:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.408083:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408097:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408099:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002a74076a[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408101:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000013a93b91 17ffffc0001001 3:0 ffff9d818877dd20 33 lru 00000080:00200000:2.0:1775764923.408104:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f6d8f811 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408105:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002a74076a 00000080:00200000:2.0:1775764923.408106:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.408107:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408114:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408116:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000038907d0b[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408117:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b3465b15 17ffffc0001001 3:0 ffff9d818877d7e0 34 lru 00000080:00200000:2.0:1775764923.408120:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f8a20af3 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408121:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000038907d0b 00000080:00200000:2.0:1775764923.408122:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.408123:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408129:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408131:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fcd0293b[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408133:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000012cfd442 17ffffc0001001 3:0 ffff9d818877d8c0 35 lru 00000080:00200000:2.0:1775764923.408135:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000091878d0f 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408136:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fcd0293b 00000080:00200000:2.0:1775764923.408137:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.408138:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408145:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408147:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000af810d31[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408148:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000094d3c7e 17ffffc0001001 3:0 ffff9d818877dc40 36 lru 00000080:00200000:2.0:1775764923.408151:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000446da001 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408152:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000af810d31 00000080:00200000:2.0:1775764923.408153:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.408154:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408160:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408162:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003fc5b7a2[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408164:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d7c63a86 17ffffc0001001 3:0 ffff9d818877da80 37 lru 00000080:00200000:2.0:1775764923.408167:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dfb75db0 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408168:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003fc5b7a2 00000080:00200000:2.0:1775764923.408180:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.408182:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408191:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408193:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000082e510c8[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408195:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a6cd7730 17ffffc0001001 3:0 ffff9d818877d380 38 lru 00000080:00200000:2.0:1775764923.408198:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000022da3d35 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408200:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000082e510c8 00000080:00200000:2.0:1775764923.408200:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.408201:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408211:0:34105:0:(rw26.c:858:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408213:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fa49c074[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408214:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000eccce85c 17ffffc0001021 2:0 ffff9d818877de00 39 lru 00000080:00200000:2.0:1775764923.408217:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fe2280fe 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408219:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fa49c074 00000080:00200000:2.0:1775764923.408219:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1775764923.408220:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408228:0:34105:0:(rw26.c:858:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408230:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b2512c6a[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408231:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a083031b 17ffffc0001001 3:0 ffff9d818877d1c0 40 lru 00000080:00200000:2.0:1775764923.408234:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c01988e8 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408236:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b2512c6a 00000080:00200000:2.0:1775764923.408236:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1775764923.408238:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408250:0:34105:0:(rw26.c:858:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408252:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000034b88ff8[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408254:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c968aac1 17ffffc0001001 3:0 ffff9d818581f7e0 41 lru 00000080:00200000:2.0:1775764923.408257:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f258c11a 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408258:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000034b88ff8 00000080:00200000:2.0:1775764923.408259:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1775764923.408260:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408267:0:34105:0:(rw26.c:858:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408269:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000017d74966[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408270:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000057691bbb 17ffffc0001001 3:0 ffff9d818581f8c0 42 lru 00000080:00200000:2.0:1775764923.408273:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000091c93e6 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408274:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000017d74966 00000080:00200000:2.0:1775764923.408275:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1775764923.408276:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408284:0:34105:0:(rw26.c:858:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408286:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000031e48333[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408287:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003b09e3ea 17ffffc0001001 3:0 ffff9d818581fc40 43 lru 00000080:00200000:2.0:1775764923.408290:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000093599403 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408291:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000031e48333 00000080:00200000:2.0:1775764923.408292:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1775764923.408293:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408299:0:34105:0:(rw26.c:858:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408301:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e63841d5[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408303:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000661ff0a6 17ffffc0001001 3:0 ffff9d818581fa80 44 lru 00000080:00200000:2.0:1775764923.408306:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c2d7735f 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408307:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e63841d5 00000080:00200000:2.0:1775764923.408307:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1775764923.408308:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408315:0:34105:0:(rw26.c:858:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408317:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fa3f4df5[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408318:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000088ad6bc5 17ffffc0001001 3:0 ffff9d818581f380 45 lru 00000080:00200000:2.0:1775764923.408321:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b542542f 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408322:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fa3f4df5 00000080:00200000:2.0:1775764923.408322:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1775764923.408323:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408330:0:34105:0:(rw26.c:858:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408332:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000097500855[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408333:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005f61e2ec 17ffffc0001001 3:0 ffff9d818581fe00 46 lru 00000080:00200000:2.0:1775764923.408336:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009ffac20f 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408337:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000097500855 00000080:00200000:2.0:1775764923.408337:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1775764923.408338:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408346:0:34105:0:(rw26.c:858:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408348:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000045ba5abc[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408349:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000040b4eed8 17ffffc0001001 3:0 ffff9d818581f1c0 47 lru 00000080:00200000:2.0:1775764923.408352:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d8e3a9e0 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408353:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000045ba5abc 00000080:00200000:2.0:1775764923.408354:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1775764923.408355:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408361:0:34105:0:(rw26.c:858:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408363:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ac3781f8[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408364:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000394c4e29 17ffffc0001001 3:0 ffff9d818581f700 48 lru 00000080:00200000:2.0:1775764923.408367:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000054050fec 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408368:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ac3781f8 00000080:00200000:2.0:1775764923.408369:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1775764923.408370:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408376:0:34105:0:(rw26.c:858:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408378:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000096c2f915[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408380:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f6bd9d66 17ffffc0001001 3:0 ffff9d818581f620 49 lru 00000080:00200000:2.0:1775764923.408383:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b590e108 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408384:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000096c2f915 00000080:00200000:2.0:1775764923.408385:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1775764923.408386:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408393:0:34105:0:(rw26.c:858:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408395:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e88767fa[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408396:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fa921621 17ffffc0001001 3:0 ffff9d818581f460 50 lru 00000080:00200000:2.0:1775764923.408399:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000edc3c3d 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408400:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e88767fa 00000080:00200000:2.0:1775764923.408400:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1775764923.408401:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408408:0:34105:0:(rw26.c:858:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408410:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f1b1fc25[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408411:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001076bd4d 17ffffc0001001 3:0 ffff9d818581f0e0 51 lru 00000080:00200000:2.0:1775764923.408414:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001f753f89 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408415:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f1b1fc25 00000080:00200000:2.0:1775764923.408415:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1775764923.408416:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 52 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408422:0:34105:0:(rw26.c:858:ll_write_end()) pos 212992, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408424:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a6e6fbdc[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408426:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c125e43d 17ffffc0001001 3:0 ffff9d818581fb60 52 lru 00000080:00200000:2.0:1775764923.408428:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a7b82805 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408430:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a6e6fbdc 00000080:00200000:2.0:1775764923.408430:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1775764923.408431:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 53 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408437:0:34105:0:(rw26.c:858:ll_write_end()) pos 217088, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408439:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009d3e8236[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408441:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000113c1a09 17ffffc0001001 3:0 ffff9d818581f9a0 53 lru 00000080:00200000:2.0:1775764923.408444:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000076b3d0f6 53: 1< 0 - - > 2< 217088 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408445:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009d3e8236 00000080:00200000:2.0:1775764923.408445:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1775764923.408447:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 54 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408456:0:34105:0:(rw26.c:858:ll_write_end()) pos 221184, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408458:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b91d00f3[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408459:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004d037a34 17ffffc0001021 2:0 ffff9d818581f540 54 lru 00000080:00200000:2.0:1775764923.408463:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000803070f3 54: 1< 0 - - > 2< 221184 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408464:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b91d00f3 00000080:00200000:2.0:1775764923.408464:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 55. 00000080:00200000:2.0:1775764923.408465:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 55 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408471:0:34105:0:(rw26.c:858:ll_write_end()) pos 225280, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408473:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002a79c872[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408475:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000277906bd 17ffffc0001001 3:0 ffff9d818581f000 55 lru 00000080:00200000:2.0:1775764923.408478:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005002dd24 55: 1< 0 - - > 2< 225280 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408479:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002a79c872 00000080:00200000:2.0:1775764923.408480:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 56. 00000080:00200000:2.0:1775764923.408480:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 56 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408487:0:34105:0:(rw26.c:858:ll_write_end()) pos 229376, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408489:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000023efb600[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408490:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f5077e97 17ffffc0001001 3:0 ffff9d818581f2a0 56 lru 00000080:00200000:2.0:1775764923.408493:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000097945962 56: 1< 0 - - > 2< 229376 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408494:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000023efb600 00000080:00200000:2.0:1775764923.408495:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 57. 00000080:00200000:2.0:1775764923.408496:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 57 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408502:0:34105:0:(rw26.c:858:ll_write_end()) pos 233472, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408504:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000063fd8fcd[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408505:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000028455b3d 17ffffc0001001 3:0 ffff9d818581fee0 57 lru 00000080:00200000:2.0:1775764923.408508:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a66890d1 57: 1< 0 - - > 2< 233472 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408509:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000063fd8fcd 00000080:00200000:2.0:1775764923.408509:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 58. 00000080:00200000:2.0:1775764923.408510:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 58 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.408516:0:34105:0:(rw26.c:858:ll_write_end()) pos 237568, len 4096, copied 4096 00000080:00200000:2.0:1775764923.408518:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002b03828f[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408520:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002a90e5ae 17ffffc0001001 3:0 ffff9d818581fd20 58 lru 00000080:00200000:2.0:1775764923.408523:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000957e5690 58: 1< 0 - - > 2< 237568 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408524:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002b03828f 00000080:00200000:2.0:1775764923.408524:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 59. 00000080:00200000:2.0:1775764923.408525:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 59 of 0 to 3224 bytes 00000080:00200000:2.0:1775764923.408539:0:34105:0:(rw26.c:858:ll_write_end()) pos 241664, len 3224, copied 3224 00000080:00200000:2.0:1775764923.408541:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008faf6eb6[3 000000005220f0eb 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.408543:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000833d863f 17ffffc0001001 3:0 ffff9d8186f34620 59 lru 00000080:00200000:2.0:1775764923.408545:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000032190063 59: 1< 0 - - > 2< 241664 0 0 0x0 0x0 | 000000009452db5f 000000007c85a8d0 > 3< 0 0 > 4< 0 0 8 2072576 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.408546:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008faf6eb6 00000080:00200000:2.0:1775764923.408547:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 60. 00000080:00200000:2.0:1775764923.408549:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 60, from 0, to 3224 prio 0 00000008:00010000:2.0:1775764923.408560:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408570:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408575:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408579:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408584:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408587:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408595:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408600:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408604:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408608:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408612:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408616:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408620:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408624:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408628:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.408636:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f5d38, count 15, dirtied 15 00000008:00010000:2.0:1775764923.408639:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408644:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408649:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408653:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408657:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408661:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408665:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408669:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408677:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408681:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408685:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408689:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408693:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408697:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408701:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.408707:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f5d38, count 15, dirtied 15 00000008:00010000:2.0:1775764923.408709:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408717:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408721:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408725:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408729:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408733:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408737:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408741:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408745:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408749:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408754:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408758:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408762:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408766:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408770:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.408776:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f5d38, count 15, dirtied 15 00000008:00010000:2.0:1775764923.408778:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408782:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408786:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408793:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408797:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408801:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408805:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408809:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408813:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408817:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408821:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408825:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408830:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408834:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.408838:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000084cc6c52 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.408844:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f5d38, count 15, dirtied 15 00000080:00200000:2.0:1775764923.408846:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 60 pages 244888 bytes, tot: 244888 00000080:00200000:2.0:1775764923.408848:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-quota.sh: write bytes 0, result: 244888 00000020:00010000:2.0:1775764923.408854:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.408856:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.408857:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ceeb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.408860:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.408864:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.408870:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.408871:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ceeb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.408889:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 244888 bytes 00000080:00200000:2.0:1775764923.408892:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 59] 00000080:00200000:2.0:1775764923.408904:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x44:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.408907:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-quota.sh: 2 io complete with rc: 0, result: 244888, restart: 0 00000080:00200000:2.0:1775764923.408911:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x44:0x0] updating i_size 244888 i_blocks 0 00000080:00200000:2.0:1775764923.408914:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 244888 00000080:00200008:2.0:1775764923.408930:0:34105:0:(file.c:447:ll_file_release()) START file sanity-quota.sh:[0x240004281:0x44:0x0](ffff9d81843f5b88), flags 100001 00000002:00100000:2.0:1775764923.408948:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fe300 x1862023066212224/t38654705893(38654705893) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.408954:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.408957:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fc380 x1862023066212864/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.408966:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fc380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066212864:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.409003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1a80 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.409670:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fc380 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066212864:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.409680:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fe300 x1862023066212224/t38654705893(38654705893) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.409703:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.409709:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.409715:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.409721:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.409725:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.409728:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.409729:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.409732:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.409734:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.409739:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.409744:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.409747:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.409749:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.409750:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.409752:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.409754:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-quota.sh, flags=8 00000080:00200008:2.0:1775764923.409757:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-quota.sh:[0x240004281:0x44:0x0](ffff9d81843f5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.409760:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x44:0x0](ffff9d81843f5b88),name=sanity-quota.sh 00000002:00010000:2.0:1775764923.409765:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x44:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.409768:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.409773:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.409777:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.409778:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x44:0x0](ffff9d81843f5b88) 00000080:00010000:2.0:1775764923.409780:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x44:0x0](ffff9d81843f5b88) for lock 0xeef44110eff567e 00000080:00010000:2.0:1775764923.409781:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff567e from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.409783:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.409790:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.409804:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x44:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.409808:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x44:0x0] 00000008:00010000:2.0:1775764923.409817:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.409818:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ceeb0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.409824:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.409830:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.409835:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.409837:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.409841:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000022eec9e7/0xeef44110eff568c lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->245759) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155993 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.409845:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.409848:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x44:0x0] updating i_size 244888 i_blocks 0 00000020:00010000:2.0:1775764923.409850:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.409852:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.409853:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ceeb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.409855:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.409857:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ceeb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.409865:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x44:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.409923:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.409929:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.409934:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.409939:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.409943:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.409946:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.409947:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.409949:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.409951:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.409955:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.409960:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.409962:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.409964:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.409966:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.409967:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.409972:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-scrub.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.409975:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-scrub.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.409982:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-scrub.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.410002:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410009:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410017:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.410028:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fea00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066212992:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.410063:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1a80 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.410799:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fea00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066212992:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.410806:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410811:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410815:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410818:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410822:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410825:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583aa00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.410827:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410833:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410837:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410840:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.410843:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.410846:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5693 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.410852:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fea00 x1862023066212992/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.410871:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.410910:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.410929:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.410934:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.410938:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.410943:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.410947:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.410949:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.410951:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.410953:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.410956:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.410960:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.410966:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.410969:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.410971:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.410972:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.410974:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.410979:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-scrub.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.410982:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-scrub.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.410989:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-scrub.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.411007:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411013:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411020:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.411031:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fea00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066213120:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.411064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1a80 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.411862:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fea00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066213120:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.411871:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411902:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411908:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411912:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411916:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411919:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583be00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.411921:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411924:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411927:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411931:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.411933:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.411937:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff569a lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.411944:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fea00 x1862023066213120/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.411954:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.411961:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.411993:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.412001:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.412008:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.412016:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.412022:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.412026:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.412028:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.412032:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.412035:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.412042:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.412050:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.412055:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.412058:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.412060:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.412061:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.412067:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.412069:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-scrub.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da000, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.412073:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-scrub.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.412082:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-scrub.sh,[0x240004281:0x45:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.412099:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.412103:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.412107:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.412111:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fea00 x1862023066213376/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.412115:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.412122:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fea00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066213376:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.412197:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc1a80 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.413183:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fea00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066213376:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.413192:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413200:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x45:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413204:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413210:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413222:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413227:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413232:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413235:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.413240:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fea00 x1862023066213376/t38654705895(38654705895) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.413246:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fea00 x1862023066213376/t38654705895(38654705895) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.413252:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.413257:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413262:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/69 (0 0) 00000080:00200000:2.0:1775764923.413264:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.413269:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-scrub.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.413290:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x45:0x0](ffff9d81843fec08) 00000080:00200000:2.0:1775764923.413355:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x45:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.413365:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x45:0x0](ffff9d81843fec08) 00000080:00010000:2.0:1775764923.413368:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x45:0x0](ffff9d81843fec08) for lock 0xeef44110eff56a1 00000080:00010000:2.0:1775764923.413374:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x45:0x0](ffff9d81843fec08) 00000080:00200008:2.0:1775764923.413378:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-scrub.sh:[0x240004281:0x45:0x0](ffff9d81843fec08), flags 100301 00000002:00100000:2.0:1775764923.413388:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fea00 x1862023066213376/t38654705895(38654705895) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.413408:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56a1 from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.413410:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413415:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.413423:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x45:0x0](ffff9d81843fec08), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.413430:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-scrub.sh:[0x240004281:0x45:0x0](ffff9d81843fec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.413432:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x45:0x0](ffff9d81843fec08),name=sanity-scrub.sh 00000002:00010000:2.0:1775764923.413440:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x45:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.413446:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.413453:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.413460:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.413464:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x45:0x0](ffff9d81843fec08) 00000080:00010000:2.0:1775764923.413466:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x45:0x0](ffff9d81843fec08) for lock 0xeef44110eff56a1 00000080:00010000:2.0:1775764923.413469:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56a1 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.413472:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.413479:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.413500:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x45:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.413505:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x45:0x0] 00000008:00010000:2.0:1775764923.413515:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.413517:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce1b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.413541:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.413547:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.413566:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.413591:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.413601:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066213504:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.414333:0:10946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.414346:0:10946:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.414354:0:10946:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.414361:0:10946:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.414368:0:10946:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.414376:0:10946:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.414386:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.414400:0:10946:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.414407:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x45:0x0] updating i_size 0 i_blocks 0 00010000:00010000:1.0:1775764923.414408:0:10946:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1775764923.414410:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.414416:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.414420:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce1b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:1.0:1775764923.414425:0:10946:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.414431:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.414435:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce1b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000100:00100000:1.0:1775764923.414448:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066213504:192.168.202.105@tcp:101:cp.0 00000080:00200000:2.0:1775764923.414450:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x45:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.414492:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-scrub.sh:[0x240004281:0x45:0x0], ppos: 0, count: 42517 00000080:00200000:2.0:1775764923.414502:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-scrub.sh: write ppos: 0, bytes: 42517 00000080:00200000:2.0:1775764923.414504:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-scrub.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.414507:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@000000000cd3db4c[0x110, 1, [0x240004281:0x45:0x0] hash] 00000020:00200000:2.0:1775764923.414509:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 42517) 0 0 00000080:00200000:2.0:1775764923.414511:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x45:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.414515:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 10] 00020000:00200000:2.0:1775764923.414517:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 42517) 42517, 42517 00020000:00200000:2.0:1775764923.414520:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.414528:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 10] 00000008:00010000:2.0:1775764923.414533:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.414535:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce1b0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.414540:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/547 (0 45055) 00010000:00010000:2.0:1775764923.414552:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.414557:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->45055] (req 0->45055) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.414562:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->45055] (req 0->45055) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.414578:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fdf80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066213632:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.414610:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc19c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.415237:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082329088 00000100:00100000:1.0:1775764923.415241:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082329088 00000100:00100000:1.0:1775764923.415268:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 291 00000100:00100000:1.0:1775764923.415276:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901af800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082329088:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.415285:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.415323:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.415328:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901af800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31386:x1862023082329088:12345-192.168.202.105@tcp:104: Request processed in 53us (130us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.415333:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 291 00010000:00010000:2.0:1775764923.415344:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.415350:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.415356:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583b600 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.415359:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.415381:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x45:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.415386:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x45:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.415411:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.415416:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.415434:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.415439:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000ca423c44/0xeef44110eff56a8 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa321559e7 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.415448:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.415449:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.415451:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.415459:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775764923.415472:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066213760:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.415937:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fdf80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066213632:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.415945:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->45055] (req 0->45055) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.415951:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->45055] (req 0->45055) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.415956:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->45055] (req 0->45055) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.416399:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.416409:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.416416:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ad180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19105:x1862023082329216:12345-192.168.202.105@tcp:105: Request processed in 102us (191us total) trans 0 rc 0/0 00010000:00010000:2.0:1775764923.416418:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416435:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.416439:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 292 00010000:00010000:2.0:1775764923.416451:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.416459:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.416476:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-scrub.sh: write [0, 42517) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.416484:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416509:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416516:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008d102ff9[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416519:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000003197c1e 17ffffc0001001 3:0 ffff9d8186f34460 0 lru 00000080:00200000:2.0:1775764923.416526:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004387a28e 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416528:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008d102ff9 00000080:00200000:2.0:1775764923.416529:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.416532:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416547:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416550:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000eb178cb[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416552:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ae846cb1 17ffffc0001001 3:0 ffff9d8186f340e0 1 lru 00000080:00200000:2.0:1775764923.416556:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001952f996 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416557:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000eb178cb 00000080:00200000:2.0:1775764923.416558:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.416559:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416566:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416568:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008af43740[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416570:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008c68a7bc 17ffffc0001001 3:0 ffff9d8186f34b60 2 lru 00000080:00200000:2.0:1775764923.416573:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e055122f 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416574:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008af43740 00000080:00200000:2.0:1775764923.416575:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.416576:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416582:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416584:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001db7fa8e[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416586:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000042438e4a 17ffffc0001001 3:0 ffff9d8186f349a0 3 lru 00000080:00200000:2.0:1775764923.416589:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000099a3aa86 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416591:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001db7fa8e 00000080:00200000:2.0:1775764923.416591:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.416592:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416598:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416600:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001971a4b9[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416602:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003b0ba54e 17ffffc0001001 3:0 ffff9d8186f34540 4 lru 00000080:00200000:2.0:1775764923.416605:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009b3d3c89 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416606:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001971a4b9 00000080:00200000:2.0:1775764923.416606:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.416607:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416614:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416616:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000043f46a17[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416617:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000018f7f40c 17ffffc0001001 3:0 ffff9d8186f34000 5 lru 00000080:00200000:2.0:1775764923.416621:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000002d4ab0e 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416622:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000043f46a17 00000080:00200000:2.0:1775764923.416623:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.416624:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416630:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416632:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dbd734ba[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416634:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004d78e55c 17ffffc0001001 3:0 ffff9d8186f342a0 6 lru 00000080:00200000:2.0:1775764923.416637:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f2cec6e6 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416638:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dbd734ba 00000080:00200000:2.0:1775764923.416638:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.416639:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416645:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416647:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000023cb1b95[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416649:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d3205fbe 17ffffc0001001 3:0 ffff9d8186f34ee0 7 lru 00000080:00200000:2.0:1775764923.416652:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d30d0700 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416653:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000023cb1b95 00000080:00200000:2.0:1775764923.416653:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.416654:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416661:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416663:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000f3f0d7f[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416665:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000094eead06 17ffffc0001001 3:0 ffff9d8186f34d20 8 lru 00000080:00200000:2.0:1775764923.416668:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004bb4f163 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416669:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000f3f0d7f 00000080:00200000:2.0:1775764923.416670:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.416671:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.416688:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.416691:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008a66306f[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416692:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e9f1814c 17ffffc0001021 2:0 ffff9d8186f347e0 9 lru 00000080:00200000:2.0:1775764923.416695:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000022154aec 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416696:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008a66306f 00000080:00200000:2.0:1775764923.416697:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.416698:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 1557 bytes 00000080:00200000:2.0:1775764923.416708:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 1557, copied 1557 00000080:00200000:2.0:1775764923.416710:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000ea0a54e[3 0000000002214f8a 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.416711:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005303ee61 17ffffc0001001 3:0 ffff9d8186f348c0 10 lru 00000080:00200000:2.0:1775764923.416714:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009f77011d 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 00000000409b8422 0000000078298771 > 3< 0 0 > 4< 0 0 8 3043328 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.416716:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000ea0a54e 00000080:00200000:2.0:1775764923.416717:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.416719:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 11, from 0, to 1557 prio 0 00000008:00010000:2.0:1775764923.416727:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416736:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416741:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416746:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416750:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416754:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416758:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416762:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416766:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416771:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.416775:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000008ecf2193 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.416781:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fedb8, count 11, dirtied 11 00000080:00200000:2.0:1775764923.416784:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 11 pages 42517 bytes, tot: 42517 00000080:00200000:2.0:1775764923.416786:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-scrub.sh: write bytes 0, result: 42517 00000020:00010000:2.0:1775764923.416791:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.416793:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.416795:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce1b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.416797:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.416802:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.416808:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.416810:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce1b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.416813:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 42517 bytes 00000080:00200000:2.0:1775764923.416815:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 10] 00000080:00200000:2.0:1775764923.416824:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x45:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.416826:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-scrub.sh: 2 io complete with rc: 0, result: 42517, restart: 0 00000080:00200000:2.0:1775764923.416829:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x45:0x0] updating i_size 42517 i_blocks 0 00000080:00200000:2.0:1775764923.416831:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 42517 00000080:00200008:2.0:1775764923.416846:0:34105:0:(file.c:447:ll_file_release()) START file sanity-scrub.sh:[0x240004281:0x45:0x0](ffff9d81843fec08), flags 100001 00000002:00100000:2.0:1775764923.416864:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fea00 x1862023066213376/t38654705895(38654705895) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.416871:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.416892:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fdf80 x1862023066214016/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.416903:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fdf80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214016:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.416939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc19c0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.416980:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066213760:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.417591:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fdf80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214016:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.417605:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fea00 x1862023066213376/t38654705895(38654705895) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.417632:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.417639:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.417645:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.417651:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.417655:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.417658:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.417660:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.417662:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.417664:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.417672:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.417677:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.417680:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.417682:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.417684:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.417686:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.417687:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-scrub.sh, flags=8 00000080:00200008:2.0:1775764923.417690:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-scrub.sh:[0x240004281:0x45:0x0](ffff9d81843fec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.417693:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x45:0x0](ffff9d81843fec08),name=sanity-scrub.sh 00000002:00010000:2.0:1775764923.417697:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x45:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.417701:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.417705:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.417709:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.417711:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x45:0x0](ffff9d81843fec08) 00000080:00010000:2.0:1775764923.417712:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x45:0x0](ffff9d81843fec08) for lock 0xeef44110eff56a1 00000080:00010000:2.0:1775764923.417714:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56a1 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.417715:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.417719:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.417733:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x45:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.417736:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x45:0x0] 00000008:00010000:2.0:1775764923.417745:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.417746:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce1b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.417752:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.417758:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.417763:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.417765:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.417769:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000c2d6d399/0xeef44110eff56af lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->45055) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321559ee expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.417773:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.417776:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x45:0x0] updating i_size 42517 i_blocks 0 00000020:00010000:2.0:1775764923.417778:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.417779:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.417781:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce1b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.417783:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.417784:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce1b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.417791:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x45:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.417815:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.417820:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.417824:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.417829:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.417833:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.417835:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.417837:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.417838:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.417840:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.417845:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.417849:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.417851:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.417852:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.417854:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.417855:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.417861:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-sec.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.417863:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-sec.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.417867:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-sec.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.417897:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.417902:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.417907:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.417915:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214144:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.417944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc19c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.418705:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214144:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.418711:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418716:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418720:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418723:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418727:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418730:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583a000 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.418732:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418735:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418739:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418742:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.418745:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418748:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56b6 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.418754:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fd500 x1862023066214144/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.418760:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.418768:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.418784:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.418788:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.418792:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.418797:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.418801:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.418803:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.418805:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.418807:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.418808:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.418812:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.418818:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.418820:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.418822:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.418824:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.418825:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.418829:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-sec.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.418831:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-sec.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.418835:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-sec.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.418846:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418850:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.418855:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.418861:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214272:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.418910:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc19c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.419632:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214272:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.419638:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419643:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419646:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419650:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419654:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419657:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d81903d8a00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.419658:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419661:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419665:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419668:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.419670:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419674:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8a00/0xeef44110eff56bd lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.419679:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fd500 x1862023066214272/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.419685:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.419691:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.419718:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.419722:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.419726:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.419732:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.419736:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.419738:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.419739:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.419741:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.419743:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.419747:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.419752:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.419754:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.419756:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.419758:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.419759:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.419764:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.419766:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-sec.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851dac00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.419770:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-sec.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.419777:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-sec.sh,[0x240004281:0x46:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.419792:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.419796:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.419801:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.419805:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fd500 x1862023066214528/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.419810:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.419817:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214528:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.419867:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8187dc19c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.420835:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd500 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214528:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.420842:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.420850:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x46:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.420854:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.420859:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.420864:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.420869:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.420889:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.420894:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.420899:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fd500 x1862023066214528/t38654705897(38654705897) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.420906:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fd500 x1862023066214528/t38654705897(38654705897) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.420913:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.420918:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.420923:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/70 (0 0) 00000080:00200000:2.0:1775764923.420925:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.420929:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-sec.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.420942:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x46:0x0](ffff9d81843f9148) 00000080:00200000:2.0:1775764923.420997:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x46:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.421007:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x46:0x0](ffff9d81843f9148) 00000080:00010000:2.0:1775764923.421009:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x46:0x0](ffff9d81843f9148) for lock 0xeef44110eff56c4 00000080:00010000:2.0:1775764923.421015:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x46:0x0](ffff9d81843f9148) 00000080:00200008:2.0:1775764923.421021:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-sec.sh:[0x240004281:0x46:0x0](ffff9d81843f9148), flags 100301 00000002:00100000:2.0:1775764923.421031:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fd500 x1862023066214528/t38654705897(38654705897) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.421054:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56c4 from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.421058:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.421066:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.421077:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x46:0x0](ffff9d81843f9148), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.421087:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-sec.sh:[0x240004281:0x46:0x0](ffff9d81843f9148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.421091:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x46:0x0](ffff9d81843f9148),name=sanity-sec.sh 00000002:00010000:2.0:1775764923.421098:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x46:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.421103:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.421108:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.421112:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.421114:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x46:0x0](ffff9d81843f9148) 00000080:00010000:2.0:1775764923.421116:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x46:0x0](ffff9d81843f9148) for lock 0xeef44110eff56c4 00000080:00010000:2.0:1775764923.421118:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56c4 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.421120:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.421124:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.421138:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x46:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.421142:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x46:0x0] 00000008:00010000:2.0:1775764923.421151:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.421153:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce730, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.421189:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 3/1,0 mode: --/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.421195:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 3/1,0 mode: --/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.421212:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.421245:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066214656:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.421860:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.421870:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 6/2,0 mode: --/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.421902:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.421912:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.421922:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.421931:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.421940:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.421949:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.421957:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x46:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.421961:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.421967:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00010000:00010000:1.0:1775764923.421968:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1775764923.421969:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce730 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.421975:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00010000:00010000:1.0:1775764923.421980:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.421980:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce730 (ffff9d8191255518/1) at cl_lock_fini():72 00000100:00100000:1.0:1775764923.421993:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066214656:192.168.202.105@tcp:101:cp.0 00000080:00200000:2.0:1775764923.422002:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x46:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.422144:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-sec.sh:[0x240004281:0x46:0x0], ppos: 0, count: 378216 00000080:00200000:2.0:1775764923.422155:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-sec.sh: write ppos: 0, bytes: 378216 00000080:00200000:2.0:1775764923.422158:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-sec.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.422161:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@0000000037439e11[0x110, 1, [0x240004281:0x46:0x0] hash] 00000020:00200000:2.0:1775764923.422162:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 378216) 0 0 00000080:00200000:2.0:1775764923.422164:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x46:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.422168:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 92] 00020000:00200000:2.0:1775764923.422185:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 378216) 378216, 378216 00020000:00200000:2.0:1775764923.422187:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.422198:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 92] 00000008:00010000:2.0:1775764923.422203:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.422205:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce730, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.422210:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/548 (0 380927) 00010000:00010000:2.0:1775764923.422222:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.422228:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->380927] (req 0->380927) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.422232:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->380927] (req 0->380927) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.422241:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ff800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214784:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.422265:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765000 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.422918:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764923.422966:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.422977:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082329344 00000100:00100000:1.0:1775764923.422980:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082329344 00000100:00100000:1.0:1775764923.423013:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 293 00000100:00100000:1.0:1775764923.423021:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796f450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082329344:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.423029:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.423067:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.423073:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796f450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:21466:x1862023082329344:12345-192.168.202.105@tcp:104: Request processed in 54us (157us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.423081:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 293 00010000:00010000:2.0:1775764923.423088:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.423093:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.423099:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583aa00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.423102:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.423128:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x46:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.423135:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x46:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.423181:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 2/0,0 mode: --/PR res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.423189:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.423210:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 1/0,0 mode: --/PR res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.423216:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff56cb lrc: 0/0,0 mode: --/PR res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155a42 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.423226:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.423228:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.423232:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.423240:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066214912:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.423634:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ff800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066214784:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.423642:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->380927] (req 0->380927) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.423648:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->380927] (req 0->380927) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.423653:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->380927] (req 0->380927) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.423983:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082329472 00000100:00100000:3.0:1775764923.423987:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082329472 00000100:00100000:3.0:1775764923.424007:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 294 00000100:00100000:3.0:1775764923.424011:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901af100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19106:x1862023082329472:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.424017:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->380927] (req 0->380927) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.424023:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->380927] (req 0->380927) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.424029:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.424034:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.424039:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.424044:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.424058:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d81902b9a00) 00010000:00010000:2.0:1775764923.424085:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.424085:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901af100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19106:x1862023082329472:12345-192.168.202.105@tcp:105: Request processed in 72us (138us total) trans 0 rc 0/0 00000100:00100000:3.0:1775764923.424092:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 294 00010000:00010000:2.0:1775764923.424094:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.424104:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.424118:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.424128:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.424137:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.424151:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-sec.sh: write [0, 378216) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.424160:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424199:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424207:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ec143229[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424210:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000668048bc 17ffffc0001001 3:0 ffff9d8186f34c40 0 lru 00000080:00200000:2.0:1775764923.424217:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000005c1a0c2 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424219:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ec143229 00000080:00200000:2.0:1775764923.424220:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.424223:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424239:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424244:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000da978248[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424247:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bd1c8758 17ffffc0001001 3:0 ffff9d8186f34a80 1 lru 00000080:00200000:2.0:1775764923.424252:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cf7ee576 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424254:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000da978248 00000080:00200000:2.0:1775764923.424255:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.424257:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424266:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424270:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000008597b11[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424272:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e154671b 17ffffc0001001 3:0 ffff9d8186f34380 2 lru 00000080:00200000:2.0:1775764923.424278:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a2a49b81 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424280:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000008597b11 00000080:00200000:2.0:1775764923.424281:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.424282:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424291:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424294:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cce01d78[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424297:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c8f5a63c 17ffffc0001001 3:0 ffff9d8186f34e00 3 lru 00000080:00200000:2.0:1775764923.424302:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000033e9f771 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424304:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cce01d78 00000080:00200000:2.0:1775764923.424305:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.424306:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424325:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424328:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000018343026[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424330:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000988f8474 17ffffc0001001 3:0 ffff9d8186f341c0 4 lru 00000080:00200000:2.0:1775764923.424333:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006c67c394 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424334:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000018343026 00000080:00200000:2.0:1775764923.424335:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.424336:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424343:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424345:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000005d657f6[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424347:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e45f13b6 17ffffc0001001 3:0 ffff9d8186f34700 5 lru 00000080:00200000:2.0:1775764923.424350:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f016da98 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424351:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000005d657f6 00000080:00200000:2.0:1775764923.424352:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.424353:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424366:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424368:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fcfee60b[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424370:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000089ed035c 17ffffc0001001 3:0 ffff9d819023a380 6 lru 00000080:00200000:2.0:1775764923.424373:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000094a38729 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424374:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fcfee60b 00000080:00200000:2.0:1775764923.424375:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.424376:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424383:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424385:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009ed2fb6e[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424387:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002e5a23d4 17ffffc0001001 3:0 ffff9d819023ae00 7 lru 00000080:00200000:2.0:1775764923.424390:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003ae3e0ad 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424391:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009ed2fb6e 00000080:00200000:2.0:1775764923.424391:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.424392:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424399:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424401:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a01e2b5d[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424402:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000095cffb60 17ffffc0001001 3:0 ffff9d819023a1c0 8 lru 00000080:00200000:2.0:1775764923.424405:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007a81bbe3 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424406:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a01e2b5d 00000080:00200000:2.0:1775764923.424407:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.424407:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424414:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424416:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000029751340[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424418:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008a2a9063 17ffffc0001001 3:0 ffff9d819023a700 9 lru 00000080:00200000:2.0:1775764923.424421:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b54cd5f9 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424421:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000029751340 00000080:00200000:2.0:1775764923.424422:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.424423:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424429:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424431:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005d05ffbc[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424433:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fd63e999 17ffffc0001001 3:0 ffff9d819023a620 10 lru 00000080:00200000:2.0:1775764923.424436:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bc1c1af7 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424437:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005d05ffbc 00000080:00200000:2.0:1775764923.424437:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.424438:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424445:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424447:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000566421f2[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424448:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000071b25af8 17ffffc0001001 3:0 ffff9d819023a460 11 lru 00000080:00200000:2.0:1775764923.424452:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dc49d685 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424453:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000566421f2 00000080:00200000:2.0:1775764923.424454:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.424455:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424461:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424463:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001553b4dd[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424464:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a7bfeadd 17ffffc0001001 3:0 ffff9d819023a0e0 12 lru 00000080:00200000:2.0:1775764923.424467:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ce74313a 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424468:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001553b4dd 00000080:00200000:2.0:1775764923.424469:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.424470:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424480:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424482:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000077f916eb[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424489:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b87b98e5 17ffffc0001021 2:0 ffff9d819023ab60 13 lru 00000080:00200000:2.0:1775764923.424492:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d0d8c8ce 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424493:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000077f916eb 00000080:00200000:2.0:1775764923.424494:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.424495:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424504:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424507:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001babf23c[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424510:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e0e6ae53 17ffffc0001001 3:0 ffff9d819023a9a0 14 lru 00000080:00200000:2.0:1775764923.424514:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009d44f1ff 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424516:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001babf23c 00000080:00200000:2.0:1775764923.424517:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.424519:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424532:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424535:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000040b06fc3[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424538:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000029a701b0 17ffffc0001001 3:0 ffff9d819023a540 15 lru 00000080:00200000:2.0:1775764923.424543:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007e928f49 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424545:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000040b06fc3 00000080:00200000:2.0:1775764923.424546:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.424548:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424561:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424565:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008ad78b2b[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424567:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000166f4b32 17ffffc0001001 3:0 ffff9d819023a000 16 lru 00000080:00200000:2.0:1775764923.424573:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fd6a4faa 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424574:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008ad78b2b 00000080:00200000:2.0:1775764923.424576:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.424577:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424590:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424594:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004d47c7bf[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424597:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f8bae723 17ffffc0001001 3:0 ffff9d819023a2a0 17 lru 00000080:00200000:2.0:1775764923.424602:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000081cc00e4 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424604:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004d47c7bf 00000080:00200000:2.0:1775764923.424606:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.424608:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424621:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424625:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009b2344d5[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424628:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000912ab1ff 17ffffc0001001 3:0 ffff9d819023aee0 18 lru 00000100:00100000:1.0:1775764923.424630:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066214912:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.424638:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000036f480f2 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424641:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009b2344d5 00000080:00200000:2.0:1775764923.424643:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.424644:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424655:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424658:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006a6c3431[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424660:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004cd36b00 17ffffc0001001 3:0 ffff9d819023ad20 19 lru 00000080:00200000:2.0:1775764923.424663:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f54e56a1 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424664:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006a6c3431 00000080:00200000:2.0:1775764923.424665:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.424666:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424673:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424675:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000531d3458[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424676:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008df66e93 17ffffc0001001 3:0 ffff9d819023a7e0 20 lru 00000080:00200000:2.0:1775764923.424680:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008aa2a2ee 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424681:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000531d3458 00000080:00200000:2.0:1775764923.424682:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.424683:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424689:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424691:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000182fa3db[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424693:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000039b7b82e 17ffffc0001001 3:0 ffff9d819023a8c0 21 lru 00000080:00200000:2.0:1775764923.424696:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000004192f5e 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424697:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000182fa3db 00000080:00200000:2.0:1775764923.424698:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.424699:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424705:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424707:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000019dcbac0[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424708:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f4d71ed8 17ffffc0001001 3:0 ffff9d819023ac40 22 lru 00000080:00200000:2.0:1775764923.424712:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000091ddd86b 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424712:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000019dcbac0 00000080:00200000:2.0:1775764923.424713:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.424714:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424720:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424722:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d2b41204[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424723:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000343c83bf 17ffffc0001001 3:0 ffff9d819023aa80 23 lru 00000080:00200000:2.0:1775764923.424726:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000174fc330 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424727:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d2b41204 00000080:00200000:2.0:1775764923.424728:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.424729:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424740:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424742:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009bbf2183[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424744:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000095ca2814 17ffffc0001001 3:0 ffff9d8187678a80 24 lru 00000080:00200000:2.0:1775764923.424747:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000abfe78c7 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424748:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009bbf2183 00000080:00200000:2.0:1775764923.424749:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.424750:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424757:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424759:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c4442c5c[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424761:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002b76ff7a 17ffffc0001001 3:0 ffff9d8187678380 25 lru 00000080:00200000:2.0:1775764923.424764:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006b74a7b2 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424765:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c4442c5c 00000080:00200000:2.0:1775764923.424766:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.424767:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424773:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424775:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000039ac52ad[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424777:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000de53aa0e 17ffffc0001001 3:0 ffff9d8187678e00 26 lru 00000080:00200000:2.0:1775764923.424780:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a411704e 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424781:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000039ac52ad 00000080:00200000:2.0:1775764923.424781:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.424782:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424789:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424791:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000066bd5cbd[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424792:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000df0f2231 17ffffc0001001 3:0 ffff9d81876781c0 27 lru 00000080:00200000:2.0:1775764923.424795:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000135bacd1 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424796:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000066bd5cbd 00000080:00200000:2.0:1775764923.424796:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.424797:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424807:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424809:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b2d0cfe8[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424811:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000256bd7b3 17ffffc0001021 2:0 ffff9d8187678700 28 lru 00000080:00200000:2.0:1775764923.424820:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000df788e39 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424822:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b2d0cfe8 00000080:00200000:2.0:1775764923.424823:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.424824:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424831:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424834:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000abba1632[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424835:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ef90f313 17ffffc0001001 3:0 ffff9d8187678620 29 lru 00000080:00200000:2.0:1775764923.424839:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001435a8fb 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424840:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000abba1632 00000080:00200000:2.0:1775764923.424840:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.424841:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424848:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424850:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d85b3d6a[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424851:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a4adab1a 17ffffc0001001 3:0 ffff9d8187678460 30 lru 00000080:00200000:2.0:1775764923.424854:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000086fa3105 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424855:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d85b3d6a 00000080:00200000:2.0:1775764923.424856:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.424857:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424864:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424866:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001c2bbb1f[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424867:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003b9737f0 17ffffc0001001 3:0 ffff9d81876780e0 31 lru 00000080:00200000:2.0:1775764923.424870:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b2429256 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424872:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001c2bbb1f 00000080:00200000:2.0:1775764923.424872:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.424873:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424897:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424900:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000021a07c51[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424901:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d7c8332a 17ffffc0001001 3:0 ffff9d8187678b60 32 lru 00000080:00200000:2.0:1775764923.424905:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003acd1c36 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424906:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000021a07c51 00000080:00200000:2.0:1775764923.424907:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.424908:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424914:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424916:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a36b1846[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424918:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b6574427 17ffffc0001001 3:0 ffff9d81876789a0 33 lru 00000080:00200000:2.0:1775764923.424921:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008e26eecf 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424922:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a36b1846 00000080:00200000:2.0:1775764923.424923:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.424924:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424931:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424933:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cf47c9be[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424934:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004eb53f38 17ffffc0001001 3:0 ffff9d8187678540 34 lru 00000080:00200000:2.0:1775764923.424937:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f21abeb8 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424939:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cf47c9be 00000080:00200000:2.0:1775764923.424939:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.424940:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424948:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424950:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d05d4bf2[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424952:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000cca67ef 17ffffc0001001 3:0 ffff9d8187678000 35 lru 00000080:00200000:2.0:1775764923.424955:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b9cc9518 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424956:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d05d4bf2 00000080:00200000:2.0:1775764923.424957:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.424958:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424964:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424966:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e362608e[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424968:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e5cd03a3 17ffffc0001001 3:0 ffff9d81876782a0 36 lru 00000080:00200000:2.0:1775764923.424971:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e19a68c8 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424972:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e362608e 00000080:00200000:2.0:1775764923.424972:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.424973:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.424981:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.424984:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000305d5a8b[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.424986:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c52c11b6 17ffffc0001001 3:0 ffff9d8187678ee0 37 lru 00000080:00200000:2.0:1775764923.424991:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c888967e 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.424992:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000305d5a8b 00000080:00200000:2.0:1775764923.424993:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.424994:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425006:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425010:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002503e30d[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425013:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b8a4356b 17ffffc0001001 3:0 ffff9d8187678d20 38 lru 00000080:00200000:2.0:1775764923.425018:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e7ccdc52 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425020:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002503e30d 00000080:00200000:2.0:1775764923.425021:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.425022:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425033:0:34105:0:(rw26.c:858:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425035:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bafe018a[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425036:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000127fc7aa 17ffffc0001001 3:0 ffff9d81876787e0 39 lru 00000080:00200000:2.0:1775764923.425040:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007c0b716e 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425041:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bafe018a 00000080:00200000:2.0:1775764923.425041:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1775764923.425042:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425049:0:34105:0:(rw26.c:858:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425051:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006e72e1c9[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425052:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000eb780d81 17ffffc0001001 3:0 ffff9d81876788c0 40 lru 00000080:00200000:2.0:1775764923.425055:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006f108d5a 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425056:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006e72e1c9 00000080:00200000:2.0:1775764923.425057:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1775764923.425058:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425064:0:34105:0:(rw26.c:858:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425066:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a4d58980[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425067:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f4fdb64d 17ffffc0001001 3:0 ffff9d8187678c40 41 lru 00000080:00200000:2.0:1775764923.425070:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000058dc9aa2 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425071:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a4d58980 00000080:00200000:2.0:1775764923.425072:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1775764923.425073:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425084:0:34105:0:(rw26.c:858:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425086:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000066417180[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425087:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d9271ca4 17ffffc0001001 3:0 ffff9d8186c4f8c0 42 lru 00000080:00200000:2.0:1775764923.425090:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b978f8b1 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425091:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000066417180 00000080:00200000:2.0:1775764923.425092:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1775764923.425093:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425102:0:34105:0:(rw26.c:858:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425104:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000011089e3[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425106:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a5414cea 17ffffc0001021 2:0 ffff9d8186c4fc40 43 lru 00000080:00200000:2.0:1775764923.425109:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000caac41ea 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425110:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000011089e3 00000080:00200000:2.0:1775764923.425111:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1775764923.425111:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425118:0:34105:0:(rw26.c:858:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425120:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000024547ecf[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425121:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000058d7cdbf 17ffffc0001001 3:0 ffff9d8186c4fa80 44 lru 00000080:00200000:2.0:1775764923.425124:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dd07c76e 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425125:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000024547ecf 00000080:00200000:2.0:1775764923.425126:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1775764923.425127:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425134:0:34105:0:(rw26.c:858:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425136:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000024664bec[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425137:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001ccd10a2 17ffffc0001001 3:0 ffff9d8186c4f380 45 lru 00000080:00200000:2.0:1775764923.425140:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fda15359 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425142:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000024664bec 00000080:00200000:2.0:1775764923.425142:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1775764923.425143:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425150:0:34105:0:(rw26.c:858:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425152:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005ce56e3a[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425153:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f7f116f0 17ffffc0001001 3:0 ffff9d8186c4fe00 46 lru 00000080:00200000:2.0:1775764923.425156:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000030907ed 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425157:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005ce56e3a 00000080:00200000:2.0:1775764923.425158:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1775764923.425159:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425165:0:34105:0:(rw26.c:858:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425167:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000010ccf857[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425181:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000023a2becc 17ffffc0001001 3:0 ffff9d8186c4f1c0 47 lru 00000080:00200000:2.0:1775764923.425186:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009d848237 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425187:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000010ccf857 00000080:00200000:2.0:1775764923.425188:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1775764923.425189:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425199:0:34105:0:(rw26.c:858:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425201:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005a696a24[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425203:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f17ce03e 17ffffc0001001 3:0 ffff9d8186c4f700 48 lru 00000080:00200000:2.0:1775764923.425206:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b2889460 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425207:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005a696a24 00000080:00200000:2.0:1775764923.425208:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1775764923.425209:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425216:0:34105:0:(rw26.c:858:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425217:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f18467f2[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425219:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000074ce1c20 17ffffc0001001 3:0 ffff9d8186c4f620 49 lru 00000080:00200000:2.0:1775764923.425222:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005d3eccac 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425223:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f18467f2 00000080:00200000:2.0:1775764923.425223:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1775764923.425224:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425231:0:34105:0:(rw26.c:858:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425232:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bc1a11b8[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425234:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009db7b71d 17ffffc0001001 3:0 ffff9d8186c4f460 50 lru 00000080:00200000:2.0:1775764923.425237:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001ae12d52 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425238:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bc1a11b8 00000080:00200000:2.0:1775764923.425239:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1775764923.425240:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425246:0:34105:0:(rw26.c:858:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425248:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cf3c1b76[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425249:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000024d9d120 17ffffc0001001 3:0 ffff9d8186c4f0e0 51 lru 00000080:00200000:2.0:1775764923.425252:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000360c3885 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425253:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cf3c1b76 00000080:00200000:2.0:1775764923.425254:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1775764923.425255:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 52 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425261:0:34105:0:(rw26.c:858:ll_write_end()) pos 212992, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425263:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000adbee8fc[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425265:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004a2089c3 17ffffc0001001 3:0 ffff9d8186c4fb60 52 lru 00000080:00200000:2.0:1775764923.425268:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000035fbf000 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425269:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000adbee8fc 00000080:00200000:2.0:1775764923.425269:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1775764923.425270:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 53 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425277:0:34105:0:(rw26.c:858:ll_write_end()) pos 217088, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425279:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000991428aa[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425281:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000005614002 17ffffc0001001 3:0 ffff9d8186c4f9a0 53 lru 00000080:00200000:2.0:1775764923.425284:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009c7ee498 53: 1< 0 - - > 2< 217088 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425285:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000991428aa 00000080:00200000:2.0:1775764923.425285:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1775764923.425286:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 54 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425292:0:34105:0:(rw26.c:858:ll_write_end()) pos 221184, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425294:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000683dae4e[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425296:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006cdb62bf 17ffffc0001001 3:0 ffff9d8186c4f540 54 lru 00000080:00200000:2.0:1775764923.425299:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000624c5a31 54: 1< 0 - - > 2< 221184 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425300:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000683dae4e 00000080:00200000:2.0:1775764923.425300:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 55. 00000080:00200000:2.0:1775764923.425301:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 55 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425308:0:34105:0:(rw26.c:858:ll_write_end()) pos 225280, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425309:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000cb832b0[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425311:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f36a4685 17ffffc0001001 3:0 ffff9d8186c4f000 55 lru 00000080:00200000:2.0:1775764923.425314:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008e5b56f9 55: 1< 0 - - > 2< 225280 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425315:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000cb832b0 00000080:00200000:2.0:1775764923.425315:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 56. 00000080:00200000:2.0:1775764923.425316:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 56 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425323:0:34105:0:(rw26.c:858:ll_write_end()) pos 229376, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425325:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003fd46ac7[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425326:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b559d7ff 17ffffc0001001 3:0 ffff9d8186c4f2a0 56 lru 00000080:00200000:2.0:1775764923.425329:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000058809beb 56: 1< 0 - - > 2< 229376 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425331:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003fd46ac7 00000080:00200000:2.0:1775764923.425332:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 57. 00000080:00200000:2.0:1775764923.425332:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 57 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425339:0:34105:0:(rw26.c:858:ll_write_end()) pos 233472, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425341:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003c6ea7fb[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425342:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a2b21f4d 17ffffc0001001 3:0 ffff9d8186c4fee0 57 lru 00000080:00200000:2.0:1775764923.425345:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000697724c6 57: 1< 0 - - > 2< 233472 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425346:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003c6ea7fb 00000080:00200000:2.0:1775764923.425346:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 58. 00000080:00200000:2.0:1775764923.425347:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 58 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425356:0:34105:0:(rw26.c:858:ll_write_end()) pos 237568, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425358:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000af250a66[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425359:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f9e3a1ce 17ffffc0001021 2:0 ffff9d8186c4fd20 58 lru 00000080:00200000:2.0:1775764923.425362:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c540f19d 58: 1< 0 - - > 2< 237568 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425363:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000af250a66 00000080:00200000:2.0:1775764923.425364:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 59. 00000080:00200000:2.0:1775764923.425365:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 59 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425371:0:34105:0:(rw26.c:858:ll_write_end()) pos 241664, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425373:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001d279512[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425374:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000aeca5e97 17ffffc0001001 3:0 ffff9d8186c4f7e0 59 lru 00000080:00200000:2.0:1775764923.425377:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000071ff0c0f 59: 1< 0 - - > 2< 241664 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425378:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001d279512 00000080:00200000:2.0:1775764923.425379:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 60. 00000080:00200000:2.0:1775764923.425380:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 60 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425392:0:34105:0:(rw26.c:858:ll_write_end()) pos 245760, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425394:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008b5a1b11[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425395:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fa251f82 17ffffc0001001 3:0 ffff9d8190eea540 60 lru 00000080:00200000:2.0:1775764923.425398:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006618b0b1 60: 1< 0 - - > 2< 245760 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425399:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008b5a1b11 00000080:00200000:2.0:1775764923.425399:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 61. 00000080:00200000:2.0:1775764923.425401:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 61 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425407:0:34105:0:(rw26.c:858:ll_write_end()) pos 249856, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425409:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000061e97f57[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425410:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006c479392 17ffffc0001001 3:0 ffff9d8190eea000 61 lru 00000080:00200000:2.0:1775764923.425414:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007556784b 61: 1< 0 - - > 2< 249856 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425415:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000061e97f57 00000080:00200000:2.0:1775764923.425416:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 62. 00000080:00200000:2.0:1775764923.425416:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 62 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425423:0:34105:0:(rw26.c:858:ll_write_end()) pos 253952, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425424:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a9ea4e13[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425426:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000056e5d7dd 17ffffc0001001 3:0 ffff9d8190eea2a0 62 lru 00000080:00200000:2.0:1775764923.425429:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a28b24ec 62: 1< 0 - - > 2< 253952 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425430:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a9ea4e13 00000080:00200000:2.0:1775764923.425431:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 63. 00000080:00200000:2.0:1775764923.425432:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 63 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425438:0:34105:0:(rw26.c:858:ll_write_end()) pos 258048, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425440:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000390e562[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425442:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000010eaeb79 17ffffc0001001 3:0 ffff9d8190eeaee0 63 lru 00000080:00200000:2.0:1775764923.425444:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000003027395 63: 1< 0 - - > 2< 258048 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425445:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000390e562 00000080:00200000:2.0:1775764923.425446:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 64. 00000080:00200000:2.0:1775764923.425447:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 64 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425456:0:34105:0:(rw26.c:858:ll_write_end()) pos 262144, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425458:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002ba6ad0d[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425459:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002e3ba1d6 17ffffc0001001 3:0 ffff9d8190eead20 64 lru 00000080:00200000:2.0:1775764923.425462:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000660256d8 64: 1< 0 - - > 2< 262144 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425463:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002ba6ad0d 00000080:00200000:2.0:1775764923.425464:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 65. 00000080:00200000:2.0:1775764923.425464:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 65 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425471:0:34105:0:(rw26.c:858:ll_write_end()) pos 266240, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425473:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a77124bd[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425475:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000066fbec59 17ffffc0001001 3:0 ffff9d8190eea7e0 65 lru 00000080:00200000:2.0:1775764923.425478:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001360fccf 65: 1< 0 - - > 2< 266240 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425479:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a77124bd 00000080:00200000:2.0:1775764923.425480:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 66. 00000080:00200000:2.0:1775764923.425481:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 66 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425487:0:34105:0:(rw26.c:858:ll_write_end()) pos 270336, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425489:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004e6ee778[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425491:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000064a75edf 17ffffc0001001 3:0 ffff9d8190eea8c0 66 lru 00000080:00200000:2.0:1775764923.425494:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d4bf0419 66: 1< 0 - - > 2< 270336 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425495:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004e6ee778 00000080:00200000:2.0:1775764923.425496:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 67. 00000080:00200000:2.0:1775764923.425496:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 67 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425504:0:34105:0:(rw26.c:858:ll_write_end()) pos 274432, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425506:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c3bf5ddc[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425507:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005444a17b 17ffffc0001001 3:0 ffff9d8190eeac40 67 lru 00000080:00200000:2.0:1775764923.425510:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f06084ba 67: 1< 0 - - > 2< 274432 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425511:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c3bf5ddc 00000080:00200000:2.0:1775764923.425512:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 68. 00000080:00200000:2.0:1775764923.425513:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 68 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425519:0:34105:0:(rw26.c:858:ll_write_end()) pos 278528, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425521:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000a016abd[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425522:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000503e4440 17ffffc0001001 3:0 ffff9d8190eeaa80 68 lru 00000080:00200000:2.0:1775764923.425525:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009be3ef51 68: 1< 0 - - > 2< 278528 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425526:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000a016abd 00000080:00200000:2.0:1775764923.425527:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 69. 00000080:00200000:2.0:1775764923.425528:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 69 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425534:0:34105:0:(rw26.c:858:ll_write_end()) pos 282624, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425536:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000046abf5ae[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425538:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002ccab93d 17ffffc0001001 3:0 ffff9d8190eea380 69 lru 00000080:00200000:2.0:1775764923.425541:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000aa0c3653 69: 1< 0 - - > 2< 282624 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425541:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000046abf5ae 00000080:00200000:2.0:1775764923.425542:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 70. 00000080:00200000:2.0:1775764923.425543:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 70 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425550:0:34105:0:(rw26.c:858:ll_write_end()) pos 286720, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425551:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000646704fa[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425553:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000051ca05bc 17ffffc0001001 3:0 ffff9d8190eeae00 70 lru 00000080:00200000:2.0:1775764923.425556:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007cea0452 70: 1< 0 - - > 2< 286720 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425557:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000646704fa 00000080:00200000:2.0:1775764923.425557:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 71. 00000080:00200000:2.0:1775764923.425558:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 71 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425564:0:34105:0:(rw26.c:858:ll_write_end()) pos 290816, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425566:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c8d40616[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425568:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bc2f357d 17ffffc0001001 3:0 ffff9d8190eea1c0 71 lru 00000080:00200000:2.0:1775764923.425570:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000089efaa45 71: 1< 0 - - > 2< 290816 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425571:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c8d40616 00000080:00200000:2.0:1775764923.425572:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 72. 00000080:00200000:2.0:1775764923.425573:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 72 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425579:0:34105:0:(rw26.c:858:ll_write_end()) pos 294912, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425581:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007404197f[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425582:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006d687a3b 17ffffc0001001 3:0 ffff9d8190eea700 72 lru 00000080:00200000:2.0:1775764923.425585:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000043906c0a 72: 1< 0 - - > 2< 294912 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425586:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007404197f 00000080:00200000:2.0:1775764923.425587:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 73. 00000080:00200000:2.0:1775764923.425588:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 73 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425596:0:34105:0:(rw26.c:858:ll_write_end()) pos 299008, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425598:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ac1b52fc[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425599:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000067f312c 17ffffc0001021 2:0 ffff9d8190eea620 73 lru 00000080:00200000:2.0:1775764923.425602:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001294f999 73: 1< 0 - - > 2< 299008 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425603:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ac1b52fc 00000080:00200000:2.0:1775764923.425604:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 74. 00000080:00200000:2.0:1775764923.425605:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 74 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425612:0:34105:0:(rw26.c:858:ll_write_end()) pos 303104, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425614:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b0f191ba[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425616:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cde33f60 17ffffc0001001 3:0 ffff9d8190eea460 74 lru 00000080:00200000:2.0:1775764923.425619:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005aa6eb25 74: 1< 0 - - > 2< 303104 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425620:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b0f191ba 00000080:00200000:2.0:1775764923.425621:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 75. 00000080:00200000:2.0:1775764923.425622:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 75 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425629:0:34105:0:(rw26.c:858:ll_write_end()) pos 307200, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425630:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e0531fc1[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425632:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b8af681f 17ffffc0001001 3:0 ffff9d8190eea0e0 75 lru 00000080:00200000:2.0:1775764923.425635:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006c11a80a 75: 1< 0 - - > 2< 307200 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425636:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e0531fc1 00000080:00200000:2.0:1775764923.425637:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 76. 00000080:00200000:2.0:1775764923.425638:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 76 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425644:0:34105:0:(rw26.c:858:ll_write_end()) pos 311296, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425646:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000635b8de9[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425648:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006311d383 17ffffc0001001 3:0 ffff9d8190eeab60 76 lru 00000080:00200000:2.0:1775764923.425651:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000231198b5 76: 1< 0 - - > 2< 311296 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425652:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000635b8de9 00000080:00200000:2.0:1775764923.425653:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 77. 00000080:00200000:2.0:1775764923.425654:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 77 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425660:0:34105:0:(rw26.c:858:ll_write_end()) pos 315392, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425662:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a7ea9253[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425663:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002afa10c9 17ffffc0001001 3:0 ffff9d8190eea9a0 77 lru 00000080:00200000:2.0:1775764923.425666:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ec0578b4 77: 1< 0 - - > 2< 315392 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425667:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a7ea9253 00000080:00200000:2.0:1775764923.425668:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 78. 00000080:00200000:2.0:1775764923.425669:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 78 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425680:0:34105:0:(rw26.c:858:ll_write_end()) pos 319488, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425683:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000038522a8d[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425684:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b8a0c9dd 17ffffc0001001 3:0 ffff9d8191224620 78 lru 00000080:00200000:2.0:1775764923.425687:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000ec45e90 78: 1< 0 - - > 2< 319488 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425689:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000038522a8d 00000080:00200000:2.0:1775764923.425689:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 79. 00000080:00200000:2.0:1775764923.425690:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 79 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425697:0:34105:0:(rw26.c:858:ll_write_end()) pos 323584, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425699:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000017369569[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425700:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000041a30158 17ffffc0001001 3:0 ffff9d8191224460 79 lru 00000080:00200000:2.0:1775764923.425703:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000063758433 79: 1< 0 - - > 2< 323584 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425704:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000017369569 00000080:00200000:2.0:1775764923.425705:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 80. 00000080:00200000:2.0:1775764923.425706:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 80 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425712:0:34105:0:(rw26.c:858:ll_write_end()) pos 327680, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425714:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007de95e4c[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425715:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ce711801 17ffffc0001001 3:0 ffff9d81912240e0 80 lru 00000080:00200000:2.0:1775764923.425718:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b325d592 80: 1< 0 - - > 2< 327680 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425719:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007de95e4c 00000080:00200000:2.0:1775764923.425720:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 81. 00000080:00200000:2.0:1775764923.425720:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 81 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425728:0:34105:0:(rw26.c:858:ll_write_end()) pos 331776, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425730:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000015dfe979[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425731:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f19a4238 17ffffc0001001 3:0 ffff9d8191224b60 81 lru 00000080:00200000:2.0:1775764923.425734:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ea492e1c 81: 1< 0 - - > 2< 331776 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425735:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000015dfe979 00000080:00200000:2.0:1775764923.425735:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 82. 00000080:00200000:2.0:1775764923.425736:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 82 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425743:0:34105:0:(rw26.c:858:ll_write_end()) pos 335872, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425745:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000be15ccba[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425746:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bb2c06b6 17ffffc0001001 3:0 ffff9d81912249a0 82 lru 00000080:00200000:2.0:1775764923.425749:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000096ef83e6 82: 1< 0 - - > 2< 335872 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425750:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000be15ccba 00000080:00200000:2.0:1775764923.425751:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 83. 00000080:00200000:2.0:1775764923.425752:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 83 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425758:0:34105:0:(rw26.c:858:ll_write_end()) pos 339968, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425760:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e172e901[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425762:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006de6ae63 17ffffc0001001 3:0 ffff9d8191224540 83 lru 00000080:00200000:2.0:1775764923.425765:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000081f621e7 83: 1< 0 - - > 2< 339968 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425766:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e172e901 00000080:00200000:2.0:1775764923.425767:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 84. 00000080:00200000:2.0:1775764923.425768:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 84 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425775:0:34105:0:(rw26.c:858:ll_write_end()) pos 344064, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425776:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000081ae3311[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425778:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005b594563 17ffffc0001001 3:0 ffff9d8191224000 84 lru 00000080:00200000:2.0:1775764923.425781:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005a2d06cd 84: 1< 0 - - > 2< 344064 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425782:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000081ae3311 00000080:00200000:2.0:1775764923.425782:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 85. 00000080:00200000:2.0:1775764923.425783:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 85 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425790:0:34105:0:(rw26.c:858:ll_write_end()) pos 348160, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425791:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000706f7a32[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425793:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a57aea4d 17ffffc0001001 3:0 ffff9d81912242a0 85 lru 00000080:00200000:2.0:1775764923.425796:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b227a107 85: 1< 0 - - > 2< 348160 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425797:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000706f7a32 00000080:00200000:2.0:1775764923.425797:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 86. 00000080:00200000:2.0:1775764923.425798:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 86 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425805:0:34105:0:(rw26.c:858:ll_write_end()) pos 352256, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425807:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cc9de115[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425808:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000390edaa5 17ffffc0001001 3:0 ffff9d8191224ee0 86 lru 00000080:00200000:2.0:1775764923.425811:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005bef13e4 86: 1< 0 - - > 2< 352256 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425812:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cc9de115 00000080:00200000:2.0:1775764923.425813:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 87. 00000080:00200000:2.0:1775764923.425814:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 87 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425820:0:34105:0:(rw26.c:858:ll_write_end()) pos 356352, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425822:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000094170b25[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425823:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bbe1686e 17ffffc0001001 3:0 ffff9d8191224d20 87 lru 00000080:00200000:2.0:1775764923.425826:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004ae3d628 87: 1< 0 - - > 2< 356352 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425827:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000094170b25 00000080:00200000:2.0:1775764923.425828:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 88. 00000080:00200000:2.0:1775764923.425829:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 88 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425838:0:34105:0:(rw26.c:858:ll_write_end()) pos 360448, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425840:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000066223ec4[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425842:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ed5bd19f 17ffffc0001021 2:0 ffff9d81912247e0 88 lru 00000080:00200000:2.0:1775764923.425845:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f65a2c50 88: 1< 0 - - > 2< 360448 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425846:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000066223ec4 00000080:00200000:2.0:1775764923.425847:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 89. 00000080:00200000:2.0:1775764923.425848:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 89 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425854:0:34105:0:(rw26.c:858:ll_write_end()) pos 364544, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425856:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000083bc9b3b[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425857:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c5bd90c5 17ffffc0001001 3:0 ffff9d81912248c0 89 lru 00000080:00200000:2.0:1775764923.425860:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000675f4bbd 89: 1< 0 - - > 2< 364544 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425862:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000083bc9b3b 00000080:00200000:2.0:1775764923.425863:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 90. 00000080:00200000:2.0:1775764923.425864:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 90 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425870:0:34105:0:(rw26.c:858:ll_write_end()) pos 368640, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425872:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a6e6c030[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425873:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000b194874 17ffffc0001001 3:0 ffff9d8191224c40 90 lru 00000080:00200000:2.0:1775764923.425893:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007d5b4278 90: 1< 0 - - > 2< 368640 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425895:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a6e6c030 00000080:00200000:2.0:1775764923.425896:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 91. 00000080:00200000:2.0:1775764923.425898:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 91 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.425911:0:34105:0:(rw26.c:858:ll_write_end()) pos 372736, len 4096, copied 4096 00000080:00200000:2.0:1775764923.425915:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c02b6a39[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425918:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000005d983bb 17ffffc0001001 3:0 ffff9d8191224a80 91 lru 00000080:00200000:2.0:1775764923.425923:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000095c3a65 91: 1< 0 - - > 2< 372736 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425925:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c02b6a39 00000080:00200000:2.0:1775764923.425926:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 92. 00000080:00200000:2.0:1775764923.425928:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 92 of 0 to 1384 bytes 00000080:00200000:2.0:1775764923.425942:0:34105:0:(rw26.c:858:ll_write_end()) pos 376832, len 1384, copied 1384 00000080:00200000:2.0:1775764923.425946:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008f5537e8[3 0000000098008224 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.425949:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b7fc0d5c 17ffffc0001001 3:0 ffff9d8191224380 92 lru 00000080:00200000:2.0:1775764923.425954:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000047965703 92: 1< 0 - - > 2< 376832 0 0 0x0 0x0 | 000000009452db5f 00000000562a6004 > 3< 0 0 > 4< 0 0 8 1802240 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.425956:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008f5537e8 00000080:00200000:2.0:1775764923.425957:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 93. 00000080:00200000:2.0:1775764923.425960:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 93, from 0, to 1384 prio 0 00000008:00010000:2.0:1775764923.425973:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.425986:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.425995:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426003:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426011:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426018:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426023:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426027:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426035:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426040:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426044:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426048:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426052:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426056:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426060:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.426068:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f92f8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.426071:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426079:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426083:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426087:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426091:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426095:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426099:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426103:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426107:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426111:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426116:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426121:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426125:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426129:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426133:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.426139:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f92f8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.426141:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426145:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426149:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426157:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426161:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426165:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426179:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426184:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426189:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426193:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426197:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426201:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426206:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426210:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426214:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.426220:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f92f8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.426223:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426227:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426231:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426235:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426239:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426246:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426250:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426254:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426258:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426262:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426266:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426270:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426274:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426278:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426283:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.426289:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f92f8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.426291:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426295:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426299:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426303:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426307:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426311:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426316:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426323:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426327:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426331:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426335:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426339:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426343:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426347:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426351:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.426357:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f92f8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.426360:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426365:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426369:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426373:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426377:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426381:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426385:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426389:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426393:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426400:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426404:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426408:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426412:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426416:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426420:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.426426:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f92f8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.426428:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426432:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.426439:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000ddd9a029 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.426443:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f92f8, count 3, dirtied 3 00000080:00200000:2.0:1775764923.426445:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 93 pages 378216 bytes, tot: 378216 00000080:00200000:2.0:1775764923.426448:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-sec.sh: write bytes 0, result: 378216 00000020:00010000:2.0:1775764923.426454:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.426456:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.426458:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce730 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.426460:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.426465:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.426471:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.426472:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce730 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.426476:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 378216 bytes 00000080:00200000:2.0:1775764923.426478:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 92] 00000080:00200000:2.0:1775764923.426488:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x46:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.426490:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-sec.sh: 2 io complete with rc: 0, result: 378216, restart: 0 00000080:00200000:2.0:1775764923.426494:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x46:0x0] updating i_size 378216 i_blocks 0 00000080:00200000:2.0:1775764923.426496:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 378216 00000080:00200008:2.0:1775764923.426511:0:34105:0:(file.c:447:ll_file_release()) START file sanity-sec.sh:[0x240004281:0x46:0x0](ffff9d81843f9148), flags 100001 00000002:00100000:2.0:1775764923.426529:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fd500 x1862023066214528/t38654705897(38654705897) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.426539:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.426542:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906ff800 x1862023066215168/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.426550:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ff800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215168:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.426579:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765000 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.427303:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ff800 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215168:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.427313:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fd500 x1862023066214528/t38654705897(38654705897) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.427338:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.427344:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.427350:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.427356:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.427361:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.427364:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.427365:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.427368:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.427370:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.427374:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.427383:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.427386:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.427389:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.427391:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.427392:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.427394:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-sec.sh, flags=8 00000080:00200008:2.0:1775764923.427397:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-sec.sh:[0x240004281:0x46:0x0](ffff9d81843f9148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.427400:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x46:0x0](ffff9d81843f9148),name=sanity-sec.sh 00000002:00010000:2.0:1775764923.427404:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x46:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.427408:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.427413:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.427417:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.427418:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x46:0x0](ffff9d81843f9148) 00000080:00010000:2.0:1775764923.427420:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x46:0x0](ffff9d81843f9148) for lock 0xeef44110eff56c4 00000080:00010000:2.0:1775764923.427421:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56c4 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.427423:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.427427:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.427441:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x46:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.427445:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x46:0x0] 00000008:00010000:2.0:1775764923.427454:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.427456:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce730, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.427461:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.427467:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.427472:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.427474:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.427478:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000285de17b/0xeef44110eff56d2 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->380927) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155a49 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.427483:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.427485:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x46:0x0] updating i_size 378216 i_blocks 0 00000020:00010000:2.0:1775764923.427487:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.427489:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.427490:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce730 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.427492:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.427493:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce730 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.427500:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x46:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.427524:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.427529:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.427533:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.427538:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.427542:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.427544:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.427545:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.427548:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.427550:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.427554:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.427558:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.427560:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.427562:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.427563:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.427564:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.427569:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-selinux.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.427571:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-selinux.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.427575:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-selinux.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.427588:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.427592:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.427600:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.427607:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ff100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215296:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.427634:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765000 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.428436:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ff100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215296:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.428443:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428448:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428452:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428456:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428459:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428462:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583b600 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.428464:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428468:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428471:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428475:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.428477:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428481:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff56d9 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.428487:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906ff100 x1862023066215296/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.428493:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.428500:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.428515:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.428520:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.428523:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.428528:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.428532:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.428534:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.428536:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.428538:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.428539:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.428543:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.428549:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.428551:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.428553:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.428554:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.428556:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.428560:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity-selinux.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.428562:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-selinux.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.428566:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-selinux.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.428576:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428580:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.428584:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.428591:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ff100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215424:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.428620:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765000 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.429487:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ff100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215424:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.429493:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429497:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429501:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429505:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429508:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429511:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583be00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.429513:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429516:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429519:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429522:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.429524:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429528:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff56e0 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.429534:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906ff100 x1862023066215424/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.429539:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.429545:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.429572:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.429577:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.429580:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.429586:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.429590:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.429592:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.429593:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.429595:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.429597:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.429601:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.429606:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.429608:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.429610:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.429611:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.429613:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.429617:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.429620:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity-selinux.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da180, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.429623:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity-selinux.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.429631:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity-selinux.sh,[0x240004281:0x47:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.429646:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.429650:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.429654:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.429658:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906ff100 x1862023066215680/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.429663:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.429670:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ff100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215680:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.429727:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765000 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.430773:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ff100 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215680:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.430781:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430789:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x47:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430794:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430799:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430804:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430809:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430823:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430827:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.430832:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906ff100 x1862023066215680/t38654705899(38654705899) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.430839:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906ff100 x1862023066215680/t38654705899(38654705899) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.430845:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.430851:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430856:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/71 (0 0) 00000080:00200000:2.0:1775764923.430858:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.430862:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity-selinux.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.430874:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x47:0x0](ffff9d81843f9988) 00000080:00200000:2.0:1775764923.430937:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x47:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.430943:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x47:0x0](ffff9d81843f9988) 00000080:00010000:2.0:1775764923.430945:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x47:0x0](ffff9d81843f9988) for lock 0xeef44110eff56e7 00000080:00010000:2.0:1775764923.430948:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x47:0x0](ffff9d81843f9988) 00000080:00200008:2.0:1775764923.430953:0:34105:0:(file.c:1057:ll_file_open()) START file sanity-selinux.sh:[0x240004281:0x47:0x0](ffff9d81843f9988), flags 100301 00000002:00100000:2.0:1775764923.430961:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906ff100 x1862023066215680/t38654705899(38654705899) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.430984:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56e7 from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.430988:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.430996:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.431007:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x47:0x0](ffff9d81843f9988), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.431016:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-selinux.sh:[0x240004281:0x47:0x0](ffff9d81843f9988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.431019:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x47:0x0](ffff9d81843f9988),name=sanity-selinux.sh 00000002:00010000:2.0:1775764923.431026:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x47:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.431037:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.431042:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.431046:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.431048:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x47:0x0](ffff9d81843f9988) 00000080:00010000:2.0:1775764923.431050:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x47:0x0](ffff9d81843f9988) for lock 0xeef44110eff56e7 00000080:00010000:2.0:1775764923.431052:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56e7 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.431054:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.431059:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.431072:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x47:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.431076:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x47:0x0] 00000008:00010000:2.0:1775764923.431087:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.431089:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cef30, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.431111:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.431119:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.431144:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1775764923.432007:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.432031:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.432041:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.432059:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.432070:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.432096:0:10947:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.432107:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.432115:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.432123:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.432131:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x47:0x0] updating i_size 0 i_blocks 0 00010000:00010000:1.0:1775764923.432132:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.432136:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.432142:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.432153:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cef30 (ffff9d8191255758/1) at cl_lock_cancel():142 00000100:00100000:1.0:1775764923.432157:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066215808:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.432159:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.432166:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cef30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.432194:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x47:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.432258:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity-selinux.sh:[0x240004281:0x47:0x0], ppos: 0, count: 25608 00000080:00200000:2.0:1775764923.432269:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity-selinux.sh: write ppos: 0, bytes: 25608 00000080:00200000:2.0:1775764923.432272:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity-selinux.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.432276:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000e8c7b772[0x110, 1, [0x240004281:0x47:0x0] hash] 00000020:00200000:2.0:1775764923.432277:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 25608) 0 0 00000080:00200000:2.0:1775764923.432279:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x47:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.432283:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 6] 00020000:00200000:2.0:1775764923.432286:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 25608) 25608, 25608 00020000:00200000:2.0:1775764923.432288:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.432297:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 6] 00000008:00010000:2.0:1775764923.432302:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.432304:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cef30, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.432309:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/548 (0 28671) 00010000:00010000:2.0:1775764923.432330:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.432336:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->28671] (req 0->28671) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.432341:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->28671] (req 0->28671) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.432349:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ff480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215936:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.432371:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765c00 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.432923:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764923.432956:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082329600 00000100:00100000:1.0:1775764923.432960:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082329600 00000100:00100000:1.0:1775764923.432987:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 295 00000100:00100000:1.0:1775764923.432992:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082329600:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.432999:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.433042:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.433049:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070a300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25514:x1862023082329600:12345-192.168.202.105@tcp:104: Request processed in 56us (125us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.433057:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 295 00010000:00010000:2.0:1775764923.433066:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.433074:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.433084:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190f1a200 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.433089:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.433119:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.433126:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x47:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.433153:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.433180:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.433195:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.433201:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000088a904d7/0xeef44110eff56ee lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155a9d expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.433209:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764923.433209:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066216064:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.433212:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.433215:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.433630:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ff480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066215936:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.433637:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->28671] (req 0->28671) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.433643:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->28671] (req 0->28671) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.433648:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->28671] (req 0->28671) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.433857:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082329728 00000100:00100000:3.0:1775764923.433861:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082329728 00000100:00100000:3.0:1775764923.433907:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 296 00000100:00100000:3.0:1775764923.433915:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901ae680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:22026:x1862023082329728:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.433923:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->28671] (req 0->28671) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.433933:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->28671] (req 0->28671) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.433940:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.433945:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.433952:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.433960:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.433985:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d81903d8a00) 00010000:00010000:2.0:1775764923.434001:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.434012:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.434013:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ae680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:22026:x1862023082329728:12345-192.168.202.105@tcp:105: Request processed in 99us (203us total) trans 0 rc 0/0 00000100:00100000:3.0:1775764923.434027:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 296 00010000:00010000:2.0:1775764923.434038:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.434044:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.434051:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.434057:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.434074:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity-selinux.sh: write [0, 25608) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.434082:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.434105:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.434113:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c5d50445[3 00000000888dade3 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.434116:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ad0d0430 17ffffc0001001 3:0 ffff9d8191224e00 0 lru 00000080:00200000:2.0:1775764923.434131:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003941d998 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 000000008ed31afd > 3< 0 0 > 4< 0 0 8 2973696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.434134:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c5d50445 00000080:00200000:2.0:1775764923.434136:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.434138:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.434154:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.434157:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000eb8b188f[3 00000000888dade3 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.434159:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000acc3e329 17ffffc0001001 3:0 ffff9d81912241c0 1 lru 00000080:00200000:2.0:1775764923.434162:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ba2d4984 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 000000008ed31afd > 3< 0 0 > 4< 0 0 8 2973696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.434164:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000eb8b188f 00000080:00200000:2.0:1775764923.434165:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.434166:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.434195:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.434198:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d8efff48[3 00000000888dade3 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.434200:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e4caca91 17ffffc0001001 3:0 ffff9d8191224700 2 lru 00000080:00200000:2.0:1775764923.434203:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b4608213 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000409b8422 000000008ed31afd > 3< 0 0 > 4< 0 0 8 2973696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.434204:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d8efff48 00000080:00200000:2.0:1775764923.434205:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.434206:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.434222:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.434225:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000064980255[3 00000000888dade3 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.434228:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f34b9b81 17ffffc0001001 3:0 ffff9d81856f62a0 3 lru 00000080:00200000:2.0:1775764923.434233:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000caad189f 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000409b8422 000000008ed31afd > 3< 0 0 > 4< 0 0 8 2973696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.434235:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000064980255 00000080:00200000:2.0:1775764923.434236:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.434237:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.434251:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.434255:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009c78b4d8[3 00000000888dade3 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.434258:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000dfa34b77 17ffffc0001001 3:0 ffff9d81856f6ee0 4 lru 00000080:00200000:2.0:1775764923.434263:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000017f1b137 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000409b8422 000000008ed31afd > 3< 0 0 > 4< 0 0 8 2973696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.434265:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009c78b4d8 00000080:00200000:2.0:1775764923.434266:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.434268:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.434284:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.434288:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003f32ac28[3 00000000888dade3 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.434290:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000070164252 17ffffc0001001 3:0 ffff9d81856f6d20 5 lru 00000080:00200000:2.0:1775764923.434295:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dd1f2fde 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000409b8422 000000008ed31afd > 3< 0 0 > 4< 0 0 8 2973696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.434297:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003f32ac28 00000080:00200000:2.0:1775764923.434298:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.434300:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 1032 bytes 00000080:00200000:2.0:1775764923.434315:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 1032, copied 1032 00000080:00200000:2.0:1775764923.434319:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d1d6d5f6[3 00000000888dade3 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.434322:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000068164768 17ffffc0001001 3:0 ffff9d81856f67e0 6 lru 00000080:00200000:2.0:1775764923.434327:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000060bf3162 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 00000000409b8422 000000008ed31afd > 3< 0 0 > 4< 0 0 8 2973696 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.434329:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d1d6d5f6 00000080:00200000:2.0:1775764923.434331:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.434333:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 7, from 0, to 1032 prio 0 00000008:00010000:2.0:1775764923.434346:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 0000000001a82a5a ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.434356:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000001a82a5a ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.434361:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000001a82a5a ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.434366:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000001a82a5a ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.434370:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000001a82a5a ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.434374:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000001a82a5a ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.434378:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000001a82a5a ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.434385:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843f9b38, count 7, dirtied 7 00000080:00200000:2.0:1775764923.434388:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 7 pages 25608 bytes, tot: 25608 00000080:00200000:2.0:1775764923.434389:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity-selinux.sh: write bytes 0, result: 25608 00000020:00010000:2.0:1775764923.434396:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.434398:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.434399:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cef30 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.434402:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.434408:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.434414:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.434415:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cef30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.434418:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 25608 bytes 00000080:00200000:2.0:1775764923.434421:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 6] 00000080:00200000:2.0:1775764923.434430:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x47:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.434432:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity-selinux.sh: 2 io complete with rc: 0, result: 25608, restart: 0 00000080:00200000:2.0:1775764923.434436:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x47:0x0] updating i_size 25608 i_blocks 0 00000080:00200000:2.0:1775764923.434438:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 25608 00000080:00200008:2.0:1775764923.434453:0:34105:0:(file.c:447:ll_file_release()) START file sanity-selinux.sh:[0x240004281:0x47:0x0](ffff9d81843f9988), flags 100001 00000002:00100000:2.0:1775764923.434489:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906ff100 x1862023066215680/t38654705899(38654705899) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.434497:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.434501:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906ff480 x1862023066216320/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.434511:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ff480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066216320:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.434549:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765c00 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.434566:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066216064:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.435273:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ff480 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066216320:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.435282:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906ff100 x1862023066215680/t38654705899(38654705899) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.435355:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.435372:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.435379:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.435385:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.435389:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.435392:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.435393:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.435396:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.435398:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.435402:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.435408:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.435417:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.435421:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.435424:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.435427:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.435429:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity-selinux.sh, flags=8 00000080:00200008:2.0:1775764923.435432:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity-selinux.sh:[0x240004281:0x47:0x0](ffff9d81843f9988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.435434:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x47:0x0](ffff9d81843f9988),name=sanity-selinux.sh 00000002:00010000:2.0:1775764923.435440:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x47:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.435445:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.435450:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.435453:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.435455:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x47:0x0](ffff9d81843f9988) 00000080:00010000:2.0:1775764923.435457:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x47:0x0](ffff9d81843f9988) for lock 0xeef44110eff56e7 00000080:00010000:2.0:1775764923.435460:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff56e7 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.435463:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.435470:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.435492:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x47:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.435498:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x47:0x0] 00000008:00010000:2.0:1775764923.435512:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.435514:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cef30, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.435522:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.435531:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.435539:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.435542:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.435550:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000004e7c5713/0xeef44110eff56f5 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aa4 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.435558:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.435563:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x47:0x0] updating i_size 25608 i_blocks 0 00000020:00010000:2.0:1775764923.435566:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.435569:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.435571:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cef30 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.435574:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.435577:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cef30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.435589:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x47:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.435635:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.435641:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.435646:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.435651:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.435655:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.435658:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.435659:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.435661:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.435663:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.435671:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.435676:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.435678:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.435680:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.435681:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.435682:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.435687:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.435689:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.435694:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.435706:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.435710:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.435715:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.435722:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fed80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066216448:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.435751:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765c00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.436571:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fed80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066216448:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.436577:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436582:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436586:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436589:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436593:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436596:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583a000 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.436598:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436601:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436605:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436608:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.436610:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436614:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a000/0xeef44110eff56fc lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.436620:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fed80 x1862023066216448/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.436625:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.436632:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.436648:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.436653:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.436656:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.436661:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.436665:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.436667:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.436669:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.436671:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.436672:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.436676:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.436682:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.436684:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.436686:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.436687:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.436689:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.436693:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sanity.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.436694:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.436699:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.436709:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436716:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.436720:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.436727:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fed80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066216576:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.436753:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765c00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.437547:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fed80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066216576:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.437553:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437558:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437562:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437565:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437569:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437572:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583aa00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.437573:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437576:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437580:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437583:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.437586:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437589:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583aa00/0xeef44110eff5703 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.437594:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fed80 x1862023066216576/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.437600:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.437606:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.437633:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.437638:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.437642:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.437647:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.437651:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.437653:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.437655:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.437657:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.437659:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.437666:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.437671:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.437673:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.437675:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.437677:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.437678:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.437682:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.437685:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sanity.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851dac00, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.437689:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sanity.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.437696:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sanity.sh,[0x240004281:0x48:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.437720:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.437724:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.437729:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.437733:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fed80 x1862023066216832/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.437737:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.437745:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fed80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066216832:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.437798:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d8191765c00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.438821:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fed80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066216832:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.438828:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.438836:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 4/0,1 mode: --/PR res: [0x240004281:0x48:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.438841:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 4/0,1 mode: --/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.438846:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.438850:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.438855:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.438858:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.438863:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.438867:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fed80 x1862023066216832/t38654705901(38654705901) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.438873:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fed80 x1862023066216832/t38654705901(38654705901) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.438894:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.438901:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.438906:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/72 (0 0) 00000080:00200000:2.0:1775764923.438909:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.438913:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sanity.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.438926:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x48:0x0](ffff9d81843faa08) 00000080:00200000:2.0:1775764923.438970:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x48:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.438976:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x48:0x0](ffff9d81843faa08) 00000080:00010000:2.0:1775764923.438979:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x48:0x0](ffff9d81843faa08) for lock 0xeef44110eff570a 00000080:00010000:2.0:1775764923.438984:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x48:0x0](ffff9d81843faa08) 00000080:00200008:2.0:1775764923.438991:0:34105:0:(file.c:1057:ll_file_open()) START file sanity.sh:[0x240004281:0x48:0x0](ffff9d81843faa08), flags 100301 00000002:00100000:2.0:1775764923.439003:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fed80 x1862023066216832/t38654705901(38654705901) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.439027:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff570a from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.439029:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.439035:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.439042:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x48:0x0](ffff9d81843faa08), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.439050:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity.sh:[0x240004281:0x48:0x0](ffff9d81843faa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.439053:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x48:0x0](ffff9d81843faa08),name=sanity.sh 00000002:00010000:2.0:1775764923.439059:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x48:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.439064:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.439069:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.439073:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.439075:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x48:0x0](ffff9d81843faa08) 00000080:00010000:2.0:1775764923.439076:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x48:0x0](ffff9d81843faa08) for lock 0xeef44110eff570a 00000080:00010000:2.0:1775764923.439078:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff570a from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.439080:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.439084:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.439096:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x48:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.439099:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x48:0x0] 00000008:00010000:2.0:1775764923.439108:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.439109:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cedb0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.439133:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.439139:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.439158:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.439197:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066216960:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.439930:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.439944:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.439954:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.439969:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.439979:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.439993:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.440003:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.440011:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.440020:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.440022:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x48:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.440027:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.440032:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00010000:00010000:1.0:1775764923.440035:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.440035:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cedb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.440039:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.440043:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cedb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000100:00100000:1.0:1775764923.440056:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066216960:192.168.202.105@tcp:101:cp.0 00000080:00200000:2.0:1775764923.440061:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x48:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.440454:0:34105:0:(file.c:2612:do_file_write_iter()) START file sanity.sh:[0x240004281:0x48:0x0], ppos: 0, count: 1145249 00000080:00200000:2.0:1775764923.440465:0:34105:0:(file.c:2058:ll_file_io_generic()) sanity.sh: write ppos: 0, bytes: 1145249 00000080:00200000:2.0:1775764923.440468:0:34105:0:(file.c:1815:ll_io_set_mirror()) sanity.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.440471:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000054add9c[0x110, 1, [0x240004281:0x48:0x0] hash] 00000020:00200000:2.0:1775764923.440472:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1145249) 0 0 00000080:00200000:2.0:1775764923.440475:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x48:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.440482:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 279] 00020000:00200000:2.0:1775764923.440486:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1145249) 1145249, 1145249 00020000:00200000:2.0:1775764923.440489:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.440504:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 279] 00000008:00010000:2.0:1775764923.440510:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.440513:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cedb0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.440520:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/549 (0 1146879) 00010000:00010000:2.0:1775764923.440534:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.440539:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->1146879] (req 0->1146879) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.440544:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->1146879] (req 0->1146879) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.440553:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd880 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217088:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.440577:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81917656c0 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.441240:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764923.441286:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082329856 00000100:00100000:1.0:1775764923.441291:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082329856 00000100:00100000:1.0:1775764923.441334:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 297 00000100:00100000:1.0:1775764923.441344:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082329856:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.441354:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.441397:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.441404:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25511:x1862023082329856:12345-192.168.202.105@tcp:104: Request processed in 62us (163us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.441412:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 297 00010000:00010000:2.0:1775764923.441418:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.441425:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.441433:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190365800 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.441438:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.441463:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x48:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.441468:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x48:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.441500:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.441508:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.441527:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.441532:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff5711 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155af8 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.441542:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.441543:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.441545:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.441940:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd880 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217088:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.441948:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->1146879] (req 0->1146879) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.441953:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->1146879] (req 0->1146879) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.441958:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->1146879] (req 0->1146879) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.442271:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082329984 00000100:00100000:3.0:1775764923.442276:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082329984 00000100:00100000:3.0:1775764923.442304:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 298 00000100:00100000:3.0:1775764923.442312:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19104:x1862023082329984:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.442320:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->1146879] (req 0->1146879) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.442330:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->1146879] (req 0->1146879) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.442339:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.442348:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.442355:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.442360:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.442375:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d8190364600) 00010000:00010000:2.0:1775764923.442392:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.442402:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.442402:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901aed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19104:x1862023082329984:12345-192.168.202.105@tcp:105: Request processed in 90us (180us total) trans 0 rc 0/0 00000100:00100000:3.0:1775764923.442418:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 298 00010000:00010000:2.0:1775764923.442422:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.442433:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.442442:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.442450:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.442466:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sanity.sh: write [0, 1145249) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.442474:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442498:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442505:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004555e6a7[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442509:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000088f9bf00 17ffffc0001001 3:0 ffff9d81856f68c0 0 lru 00000080:00200000:2.0:1775764923.442514:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000db5a7017 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442517:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004555e6a7 00000080:00200000:2.0:1775764923.442518:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.442520:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442533:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442535:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005b23a243[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442537:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d49e0fe4 17ffffc0001001 3:0 ffff9d81856f6c40 1 lru 00000080:00200000:2.0:1775764923.442541:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ce0270f5 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442542:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005b23a243 00000080:00200000:2.0:1775764923.442543:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.442544:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442553:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442555:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fd371f34[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442556:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fd3a2e1a 17ffffc0001001 3:0 ffff9d81856f6a80 2 lru 00000080:00200000:2.0:1775764923.442559:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004535c99b 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442561:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fd371f34 00000080:00200000:2.0:1775764923.442561:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.442562:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442574:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442577:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ef187f3d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442579:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000007dde2fd 17ffffc0001021 2:0 ffff9d81856f6380 3 lru 00000080:00200000:2.0:1775764923.442582:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002de17774 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442583:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ef187f3d 00000080:00200000:2.0:1775764923.442584:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.442585:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442592:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442594:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000082d1dc35[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442596:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007bc503c8 17ffffc0001001 3:0 ffff9d81856f6e00 4 lru 00000080:00200000:2.0:1775764923.442599:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a60c66b9 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442600:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000082d1dc35 00000080:00200000:2.0:1775764923.442601:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.442602:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442608:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442610:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000deefd589[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442611:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000650e56ec 17ffffc0001001 3:0 ffff9d81856f61c0 5 lru 00000080:00200000:2.0:1775764923.442614:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001dbf6545 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442616:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000deefd589 00000080:00200000:2.0:1775764923.442616:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.442617:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442624:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442626:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002fa65e2e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442628:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006b4f703d 17ffffc0001001 3:0 ffff9d81856f6700 6 lru 00000080:00200000:2.0:1775764923.442631:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000066da27a9 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442632:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002fa65e2e 00000080:00200000:2.0:1775764923.442632:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.442633:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442639:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442641:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006f34e7e0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442643:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007f970219 17ffffc0001001 3:0 ffff9d81856f6620 7 lru 00000080:00200000:2.0:1775764923.442646:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009c76990e 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442647:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006f34e7e0 00000080:00200000:2.0:1775764923.442648:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.442649:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442655:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442657:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000027acf8a6[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442658:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c2ee6173 17ffffc0001001 3:0 ffff9d81856f6460 8 lru 00000080:00200000:2.0:1775764923.442661:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006914f201 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442662:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000027acf8a6 00000080:00200000:2.0:1775764923.442663:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.442664:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442671:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442673:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004c2e2da2[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442675:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001d72e290 17ffffc0001001 3:0 ffff9d81856f60e0 9 lru 00000080:00200000:2.0:1775764923.442678:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000013b670dd 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442679:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004c2e2da2 00000080:00200000:2.0:1775764923.442679:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.442680:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442687:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442689:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fcbc530c[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442690:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005ab2ef13 17ffffc0001001 3:0 ffff9d81856f6b60 10 lru 00000080:00200000:2.0:1775764923.442693:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001c81b5d9 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442694:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fcbc530c 00000080:00200000:2.0:1775764923.442695:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.442695:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442702:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442704:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002f790b50[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442706:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c2ab13ab 17ffffc0001001 3:0 ffff9d81856f69a0 11 lru 00000080:00200000:2.0:1775764923.442708:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000018ab45e1 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442709:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002f790b50 00000080:00200000:2.0:1775764923.442710:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.442711:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442717:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442719:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000012472641[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442720:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000018b7e7e7 17ffffc0001001 3:0 ffff9d81856f6540 12 lru 00000080:00200000:2.0:1775764923.442724:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d1d4fab9 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442725:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000012472641 00000080:00200000:2.0:1775764923.442726:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.442727:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442733:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442735:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a65bfa12[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442737:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a12996f6 17ffffc0001001 3:0 ffff9d81856f6000 13 lru 00000080:00200000:2.0:1775764923.442740:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007da523d4 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442741:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a65bfa12 00000080:00200000:2.0:1775764923.442742:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.442742:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442755:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442757:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000487794f2[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442758:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000068264d2a 17ffffc0001001 3:0 ffff9d8190320d20 14 lru 00000080:00200000:2.0:1775764923.442761:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009cbb3a4d 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442762:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000487794f2 00000080:00200000:2.0:1775764923.442763:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.442764:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442770:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442772:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000026c68b8e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442774:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d2ad5cbd 17ffffc0001001 3:0 ffff9d81903207e0 15 lru 00000080:00200000:2.0:1775764923.442777:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d426576f 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442778:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000026c68b8e 00000080:00200000:2.0:1775764923.442778:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.442779:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442786:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442788:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c418e6ad[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442790:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005dbfb80a 17ffffc0001001 3:0 ffff9d81903208c0 16 lru 00000080:00200000:2.0:1775764923.442793:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ce7b1a37 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442794:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c418e6ad 00000080:00200000:2.0:1775764923.442795:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.442796:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442802:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442804:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000037051384[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442806:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000edf320b9 17ffffc0001001 3:0 ffff9d8190320c40 17 lru 00000080:00200000:2.0:1775764923.442809:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003eec2a34 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442810:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000037051384 00000080:00200000:2.0:1775764923.442811:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.442812:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442821:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442823:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a2d5f39f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442825:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f4d93bcf 17ffffc0001021 2:0 ffff9d8190320a80 18 lru 00000080:00200000:2.0:1775764923.442828:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b3a2943a 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442830:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a2d5f39f 00000080:00200000:2.0:1775764923.442830:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.442831:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442839:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442842:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006fd3cb65[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442843:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000723000f9 17ffffc0001001 3:0 ffff9d8190320380 19 lru 00000080:00200000:2.0:1775764923.442847:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007408c33d 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442848:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006fd3cb65 00000080:00200000:2.0:1775764923.442849:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.442850:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442858:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442860:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000dd5df32[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442862:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008cddc727 17ffffc0001001 3:0 ffff9d8190320e00 20 lru 00000080:00200000:2.0:1775764923.442866:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c3400225 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442867:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000dd5df32 00000080:00200000:2.0:1775764923.442868:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.442869:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442899:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442905:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001dbb8cc3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442908:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000068554280 17ffffc0001001 3:0 ffff9d81903201c0 21 lru 00000080:00200000:2.0:1775764923.442913:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008bc5e864 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442916:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001dbb8cc3 00000080:00200000:2.0:1775764923.442918:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.442922:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442938:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442941:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005b58bc7d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442943:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000381d55a0 17ffffc0001001 3:0 ffff9d8190320700 22 lru 00000080:00200000:2.0:1775764923.442946:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cb6813aa 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442947:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005b58bc7d 00000080:00200000:2.0:1775764923.442948:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.442949:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442957:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442959:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000046ae2fc[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442961:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a53bf0a3 17ffffc0001001 3:0 ffff9d8190320620 23 lru 00000080:00200000:2.0:1775764923.442964:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009a73a014 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442965:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000046ae2fc 00000080:00200000:2.0:1775764923.442966:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.442967:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.442973:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.442976:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000487a56f0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.442978:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001a4f6659 17ffffc0001001 3:0 ffff9d8190320460 24 lru 00000080:00200000:2.0:1775764923.442983:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000db08a489 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.442985:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000487a56f0 00000080:00200000:2.0:1775764923.442986:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.442988:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443001:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443004:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000096031591[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443007:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009f48ae21 17ffffc0001001 3:0 ffff9d81903200e0 25 lru 00000080:00200000:2.0:1775764923.443012:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005ec052ec 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443014:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000096031591 00000080:00200000:2.0:1775764923.443015:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.443016:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443028:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443031:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ca869a76[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443032:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001561bcee 17ffffc0001001 3:0 ffff9d8190320b60 26 lru 00000080:00200000:2.0:1775764923.443036:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006ea6ad20 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443037:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ca869a76 00000080:00200000:2.0:1775764923.443038:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.443039:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443047:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443050:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a65abbe0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443053:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b1fb9821 17ffffc0001001 3:0 ffff9d81903209a0 27 lru 00000080:00200000:2.0:1775764923.443057:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f23aa356 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443059:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a65abbe0 00000080:00200000:2.0:1775764923.443060:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.443062:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443074:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443078:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b17e2653[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443081:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d94e46ff 17ffffc0001001 3:0 ffff9d8190320540 28 lru 00000080:00200000:2.0:1775764923.443086:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f212cf31 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443088:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b17e2653 00000080:00200000:2.0:1775764923.443089:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.443091:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443101:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443104:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000059eefa6d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443106:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d906d5ac 17ffffc0001001 3:0 ffff9d8190320000 29 lru 00000080:00200000:2.0:1775764923.443109:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000055504fde 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443110:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000059eefa6d 00000080:00200000:2.0:1775764923.443111:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.443112:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443120:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443128:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009de987a3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443130:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000114eee28 17ffffc0001001 3:0 ffff9d81903202a0 30 lru 00000080:00200000:2.0:1775764923.443133:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a15a201c 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443135:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009de987a3 00000080:00200000:2.0:1775764923.443135:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.443136:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443144:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443146:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000829bbb6a[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443148:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005ae694ea 17ffffc0001001 3:0 ffff9d8190320ee0 31 lru 00000080:00200000:2.0:1775764923.443151:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000068e1d037 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443152:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000829bbb6a 00000080:00200000:2.0:1775764923.443153:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.443154:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443165:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443167:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e5fc155a[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443179:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c2a156f7 17ffffc0001001 3:0 ffff9d819067b0e0 32 lru 00000080:00200000:2.0:1775764923.443183:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e8aca80d 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443184:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e5fc155a 00000080:00200000:2.0:1775764923.443185:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.443186:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443208:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443212:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fc1d067a[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443215:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000503ce60b 17ffffc0001021 2:0 ffff9d819067bb60 33 lru 00000080:00200000:2.0:1775764923.443220:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001d330f2f 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443222:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fc1d067a 00000080:00200000:2.0:1775764923.443223:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.443225:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443238:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443241:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c3889511[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443244:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002ae742d6 17ffffc0001001 3:0 ffff9d819067b9a0 34 lru 00000080:00200000:2.0:1775764923.443250:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009c624daf 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443252:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c3889511 00000080:00200000:2.0:1775764923.443253:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.443254:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443266:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443270:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003687f447[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443273:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a485acfc 17ffffc0001001 3:0 ffff9d819067b540 35 lru 00000080:00200000:2.0:1775764923.443278:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000051018974 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443280:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003687f447 00000080:00200000:2.0:1775764923.443280:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.443282:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443291:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443293:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004869f6a7[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443295:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f5659ded 17ffffc0001001 3:0 ffff9d819067b000 36 lru 00000080:00200000:2.0:1775764923.443298:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000062d6247f 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443298:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004869f6a7 00000080:00200000:2.0:1775764923.443299:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.443300:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443308:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443310:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d7668444[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443311:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b92f276e 17ffffc0001001 3:0 ffff9d819067b2a0 37 lru 00000080:00200000:2.0:1775764923.443314:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000219e6f22 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443315:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d7668444 00000080:00200000:2.0:1775764923.443315:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.443316:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443323:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443325:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006d71df4b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443326:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a7e9662e 17ffffc0001001 3:0 ffff9d819067bee0 38 lru 00000080:00200000:2.0:1775764923.443329:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008c40ad14 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443330:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006d71df4b 00000080:00200000:2.0:1775764923.443330:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.443331:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443338:0:34105:0:(rw26.c:858:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443340:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a26b3430[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443341:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001af863e6 17ffffc0001001 3:0 ffff9d819067bd20 39 lru 00000080:00200000:2.0:1775764923.443344:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e4a5e58b 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443345:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a26b3430 00000080:00200000:2.0:1775764923.443345:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1775764923.443346:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443353:0:34105:0:(rw26.c:858:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443355:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000733e16ea[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443356:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004636d1ac 17ffffc0001001 3:0 ffff9d819067b7e0 40 lru 00000080:00200000:2.0:1775764923.443359:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000022a2e2c2 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443360:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000733e16ea 00000080:00200000:2.0:1775764923.443361:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1775764923.443362:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443368:0:34105:0:(rw26.c:858:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443370:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000062e2035f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443372:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bdbdc068 17ffffc0001001 3:0 ffff9d819067b8c0 41 lru 00000080:00200000:2.0:1775764923.443374:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000036a856f0 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443375:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000062e2035f 00000080:00200000:2.0:1775764923.443376:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1775764923.443377:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443383:0:34105:0:(rw26.c:858:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443385:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000054db06ec[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443386:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000be11b339 17ffffc0001001 3:0 ffff9d819067bc40 42 lru 00000080:00200000:2.0:1775764923.443389:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000418b0e5a 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443390:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000054db06ec 00000080:00200000:2.0:1775764923.443391:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1775764923.443392:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443398:0:34105:0:(rw26.c:858:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443400:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005c575c0a[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443401:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003c12c77a 17ffffc0001001 3:0 ffff9d819067ba80 43 lru 00000080:00200000:2.0:1775764923.443404:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cb4d9675 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443405:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005c575c0a 00000080:00200000:2.0:1775764923.443406:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1775764923.443406:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443414:0:34105:0:(rw26.c:858:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443415:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c01793b6[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443417:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007094c8bf 17ffffc0001001 3:0 ffff9d819067b380 44 lru 00000080:00200000:2.0:1775764923.443420:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ea3a4f79 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443421:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c01793b6 00000080:00200000:2.0:1775764923.443422:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1775764923.443422:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443429:0:34105:0:(rw26.c:858:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443431:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001a6edc4f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443432:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a5f1d5a1 17ffffc0001001 3:0 ffff9d819067be00 45 lru 00000080:00200000:2.0:1775764923.443435:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cc51d736 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443436:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001a6edc4f 00000080:00200000:2.0:1775764923.443437:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1775764923.443438:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443444:0:34105:0:(rw26.c:858:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443446:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007c3ad82b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443448:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fd0d833b 17ffffc0001001 3:0 ffff9d819067b1c0 46 lru 00000080:00200000:2.0:1775764923.443451:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000020336cc4 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443452:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007c3ad82b 00000080:00200000:2.0:1775764923.443452:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1775764923.443453:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443459:0:34105:0:(rw26.c:858:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443461:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c6138c9e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443463:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001fcae91c 17ffffc0001001 3:0 ffff9d819067b700 47 lru 00000080:00200000:2.0:1775764923.443466:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001a970d7d 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443467:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c6138c9e 00000080:00200000:2.0:1775764923.443467:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1775764923.443468:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443479:0:34105:0:(rw26.c:858:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443483:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ea2746e4[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443485:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b355e44e 17ffffc0001021 2:0 ffff9d819067b620 48 lru 00000080:00200000:2.0:1775764923.443490:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006f362594 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443491:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ea2746e4 00000080:00200000:2.0:1775764923.443492:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1775764923.443494:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443506:0:34105:0:(rw26.c:858:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443510:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e0f8ccfd[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443512:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ea93d2ef 17ffffc0001001 3:0 ffff9d819067b460 49 lru 00000080:00200000:2.0:1775764923.443517:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e795fe91 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443519:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e0f8ccfd 00000080:00200000:2.0:1775764923.443520:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1775764923.443521:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443535:0:34105:0:(rw26.c:858:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443537:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009c01bbf5[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443538:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005e639d4b 17ffffc0001001 3:0 ffff9d8186dc61c0 50 lru 00000080:00200000:2.0:1775764923.443542:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006f3b4bd3 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443543:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009c01bbf5 00000080:00200000:2.0:1775764923.443543:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1775764923.443544:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443552:0:34105:0:(rw26.c:858:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443554:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000045ba1f84[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443555:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000be71bc83 17ffffc0001001 3:0 ffff9d8186dc6700 51 lru 00000080:00200000:2.0:1775764923.443558:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009b3f12a8 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443560:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000045ba1f84 00000080:00200000:2.0:1775764923.443560:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1775764923.443561:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 52 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443568:0:34105:0:(rw26.c:858:ll_write_end()) pos 212992, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443570:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000860dbff3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443571:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d54ac797 17ffffc0001001 3:0 ffff9d8186dc6620 52 lru 00000080:00200000:2.0:1775764923.443574:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c65e26fc 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443575:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000860dbff3 00000080:00200000:2.0:1775764923.443576:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1775764923.443577:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 53 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443584:0:34105:0:(rw26.c:858:ll_write_end()) pos 217088, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443586:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009a63e2ba[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443587:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000000187e74 17ffffc0001001 3:0 ffff9d8186dc6460 53 lru 00000080:00200000:2.0:1775764923.443590:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006a7b4bbc 53: 1< 0 - - > 2< 217088 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443592:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009a63e2ba 00000080:00200000:2.0:1775764923.443593:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1775764923.443593:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 54 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443600:0:34105:0:(rw26.c:858:ll_write_end()) pos 221184, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443602:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000048cbb2b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443603:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000049a4250f 17ffffc0001001 3:0 ffff9d8186dc60e0 54 lru 00000080:00200000:2.0:1775764923.443606:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fd247f09 54: 1< 0 - - > 2< 221184 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443607:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000048cbb2b 00000080:00200000:2.0:1775764923.443608:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 55. 00000080:00200000:2.0:1775764923.443609:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 55 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443615:0:34105:0:(rw26.c:858:ll_write_end()) pos 225280, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443617:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cf2db1fa[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443618:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b78750d0 17ffffc0001001 3:0 ffff9d8186dc6b60 55 lru 00000080:00200000:2.0:1775764923.443621:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f43c8a96 55: 1< 0 - - > 2< 225280 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443622:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cf2db1fa 00000080:00200000:2.0:1775764923.443623:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 56. 00000080:00200000:2.0:1775764923.443623:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 56 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443630:0:34105:0:(rw26.c:858:ll_write_end()) pos 229376, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443632:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005bb1cc2d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443634:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000788a2e2c 17ffffc0001001 3:0 ffff9d8186dc69a0 56 lru 00000080:00200000:2.0:1775764923.443637:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000069a23500 56: 1< 0 - - > 2< 229376 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443638:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005bb1cc2d 00000080:00200000:2.0:1775764923.443639:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 57. 00000080:00200000:2.0:1775764923.443639:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 57 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443647:0:34105:0:(rw26.c:858:ll_write_end()) pos 233472, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443649:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000098312650[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443650:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008de1f168 17ffffc0001001 3:0 ffff9d8186dc6540 57 lru 00000080:00200000:2.0:1775764923.443653:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000051fcc370 57: 1< 0 - - > 2< 233472 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443654:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000098312650 00000080:00200000:2.0:1775764923.443655:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 58. 00000080:00200000:2.0:1775764923.443656:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 58 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443663:0:34105:0:(rw26.c:858:ll_write_end()) pos 237568, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443665:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000085c6552c[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443667:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000ea2acdd 17ffffc0001001 3:0 ffff9d8186dc6000 58 lru 00000080:00200000:2.0:1775764923.443671:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000db8a53f7 58: 1< 0 - - > 2< 237568 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443673:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000085c6552c 00000080:00200000:2.0:1775764923.443674:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 59. 00000080:00200000:2.0:1775764923.443676:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 59 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443687:0:34105:0:(rw26.c:858:ll_write_end()) pos 241664, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443691:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bfc9963e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443693:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f5571544 17ffffc0001001 3:0 ffff9d8186dc62a0 59 lru 00000080:00200000:2.0:1775764923.443698:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007e120527 59: 1< 0 - - > 2< 241664 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443700:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bfc9963e 00000080:00200000:2.0:1775764923.443701:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 60. 00000080:00200000:2.0:1775764923.443703:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 60 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443715:0:34105:0:(rw26.c:858:ll_write_end()) pos 245760, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443719:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008752d8b5[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443721:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006bf19842 17ffffc0001001 3:0 ffff9d8186dc6ee0 60 lru 00000080:00200000:2.0:1775764923.443727:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d61f2d32 60: 1< 0 - - > 2< 245760 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443728:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008752d8b5 00000080:00200000:2.0:1775764923.443729:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 61. 00000080:00200000:2.0:1775764923.443730:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 61 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443739:0:34105:0:(rw26.c:858:ll_write_end()) pos 249856, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443741:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004d58709e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443743:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e390cfe9 17ffffc0001001 3:0 ffff9d8186dc6d20 61 lru 00000080:00200000:2.0:1775764923.443746:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000085ce2cfd 61: 1< 0 - - > 2< 249856 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443747:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004d58709e 00000080:00200000:2.0:1775764923.443747:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 62. 00000080:00200000:2.0:1775764923.443748:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 62 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443756:0:34105:0:(rw26.c:858:ll_write_end()) pos 253952, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443758:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000064c50605[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443760:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d98fcee2 17ffffc0001001 3:0 ffff9d8186dc67e0 62 lru 00000080:00200000:2.0:1775764923.443762:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a4ed02ef 62: 1< 0 - - > 2< 253952 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443764:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000064c50605 00000080:00200000:2.0:1775764923.443765:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 63. 00000080:00200000:2.0:1775764923.443765:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 63 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443774:0:34105:0:(rw26.c:858:ll_write_end()) pos 258048, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443776:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bce0e6ff[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443778:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000145e5360 17ffffc0001021 2:0 ffff9d8186dc68c0 63 lru 00000080:00200000:2.0:1775764923.443781:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e42725ce 63: 1< 0 - - > 2< 258048 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443782:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bce0e6ff 00000080:00200000:2.0:1775764923.443783:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 64. 00000080:00200000:2.0:1775764923.443784:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 64 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443793:0:34105:0:(rw26.c:858:ll_write_end()) pos 262144, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443795:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003c66be43[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443796:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000059dec55b 17ffffc0001001 3:0 ffff9d8186dc6c40 64 lru 00000080:00200000:2.0:1775764923.443799:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000313f6ba9 64: 1< 0 - - > 2< 262144 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443801:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003c66be43 00000080:00200000:2.0:1775764923.443801:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 65. 00000080:00200000:2.0:1775764923.443802:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 65 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443810:0:34105:0:(rw26.c:858:ll_write_end()) pos 266240, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443812:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e48007f2[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443814:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e34bdccb 17ffffc0001001 3:0 ffff9d8186dc6a80 65 lru 00000080:00200000:2.0:1775764923.443817:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001c64ee46 65: 1< 0 - - > 2< 266240 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443818:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e48007f2 00000080:00200000:2.0:1775764923.443818:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 66. 00000080:00200000:2.0:1775764923.443820:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 66 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443826:0:34105:0:(rw26.c:858:ll_write_end()) pos 270336, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443828:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000777af9aa[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443829:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004f9d19ed 17ffffc0001001 3:0 ffff9d8186dc6380 66 lru 00000080:00200000:2.0:1775764923.443833:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001845c4d0 66: 1< 0 - - > 2< 270336 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443834:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000777af9aa 00000080:00200000:2.0:1775764923.443834:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 67. 00000080:00200000:2.0:1775764923.443835:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 67 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443842:0:34105:0:(rw26.c:858:ll_write_end()) pos 274432, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443844:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d723481f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443845:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b831aa42 17ffffc0001001 3:0 ffff9d8186dc6e00 67 lru 00000080:00200000:2.0:1775764923.443848:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cf69266c 67: 1< 0 - - > 2< 274432 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443849:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d723481f 00000080:00200000:2.0:1775764923.443850:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 68. 00000080:00200000:2.0:1775764923.443851:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 68 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443862:0:34105:0:(rw26.c:858:ll_write_end()) pos 278528, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443864:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f01d0937[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443865:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000abb119a6 17ffffc0001001 3:0 ffff9d819011a1c0 68 lru 00000080:00200000:2.0:1775764923.443868:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000329b56ba 68: 1< 0 - - > 2< 278528 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443869:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f01d0937 00000080:00200000:2.0:1775764923.443870:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 69. 00000080:00200000:2.0:1775764923.443871:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 69 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443896:0:34105:0:(rw26.c:858:ll_write_end()) pos 282624, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443899:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006490b05b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443901:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004c44faf9 17ffffc0001001 3:0 ffff9d819011a700 69 lru 00000080:00200000:2.0:1775764923.443905:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b1bc114d 69: 1< 0 - - > 2< 282624 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443906:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006490b05b 00000080:00200000:2.0:1775764923.443907:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 70. 00000080:00200000:2.0:1775764923.443909:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 70 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443917:0:34105:0:(rw26.c:858:ll_write_end()) pos 286720, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443920:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002bcb4a05[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443921:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004cf13a96 17ffffc0001001 3:0 ffff9d819011a620 70 lru 00000080:00200000:2.0:1775764923.443925:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d4f1ee94 70: 1< 0 - - > 2< 286720 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443926:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002bcb4a05 00000080:00200000:2.0:1775764923.443927:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 71. 00000080:00200000:2.0:1775764923.443928:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 71 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443936:0:34105:0:(rw26.c:858:ll_write_end()) pos 290816, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443939:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cad348af[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443942:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d972bbaa 17ffffc0001001 3:0 ffff9d819011a460 71 lru 00000080:00200000:2.0:1775764923.443947:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000d60f823 71: 1< 0 - - > 2< 290816 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443949:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cad348af 00000080:00200000:2.0:1775764923.443951:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 72. 00000080:00200000:2.0:1775764923.443952:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 72 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443965:0:34105:0:(rw26.c:858:ll_write_end()) pos 294912, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443969:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009af0fb6d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.443972:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000082c56270 17ffffc0001001 3:0 ffff9d819011a0e0 72 lru 00000080:00200000:2.0:1775764923.443976:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002365e0ce 72: 1< 0 - - > 2< 294912 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.443979:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009af0fb6d 00000080:00200000:2.0:1775764923.443980:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 73. 00000080:00200000:2.0:1775764923.443982:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 73 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.443994:0:34105:0:(rw26.c:858:ll_write_end()) pos 299008, len 4096, copied 4096 00000080:00200000:2.0:1775764923.443998:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bf358878[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444001:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002848f19f 17ffffc0001001 3:0 ffff9d819011ab60 73 lru 00000080:00200000:2.0:1775764923.444006:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003afc4227 73: 1< 0 - - > 2< 299008 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444008:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bf358878 00000080:00200000:2.0:1775764923.444008:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 74. 00000080:00200000:2.0:1775764923.444010:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 74 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444022:0:34105:0:(rw26.c:858:ll_write_end()) pos 303104, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444025:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009a1dd0dc[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444028:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bfb039fb 17ffffc0001001 3:0 ffff9d819011a9a0 74 lru 00000080:00200000:2.0:1775764923.444031:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000928fed31 74: 1< 0 - - > 2< 303104 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444033:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009a1dd0dc 00000080:00200000:2.0:1775764923.444033:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 75. 00000080:00200000:2.0:1775764923.444034:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 75 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444043:0:34105:0:(rw26.c:858:ll_write_end()) pos 307200, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444045:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000016651425[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444048:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d4b0cd50 17ffffc0001001 3:0 ffff9d819011a540 75 lru 00000080:00200000:2.0:1775764923.444053:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d63539b7 75: 1< 0 - - > 2< 307200 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444055:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000016651425 00000080:00200000:2.0:1775764923.444056:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 76. 00000080:00200000:2.0:1775764923.444058:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 76 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444070:0:34105:0:(rw26.c:858:ll_write_end()) pos 311296, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444074:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005d4b1a31[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444076:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002e0b5889 17ffffc0001001 3:0 ffff9d819011a000 76 lru 00000080:00200000:2.0:1775764923.444081:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c6ae5d83 76: 1< 0 - - > 2< 311296 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444084:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005d4b1a31 00000080:00200000:2.0:1775764923.444085:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 77. 00000080:00200000:2.0:1775764923.444086:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 77 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444097:0:34105:0:(rw26.c:858:ll_write_end()) pos 315392, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444100:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c29d022e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444102:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000083e5f8ea 17ffffc0001001 3:0 ffff9d819011a2a0 77 lru 00000080:00200000:2.0:1775764923.444105:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000042bba377 77: 1< 0 - - > 2< 315392 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444106:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c29d022e 00000080:00200000:2.0:1775764923.444107:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 78. 00000080:00200000:2.0:1775764923.444108:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 78 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444117:0:34105:0:(rw26.c:858:ll_write_end()) pos 319488, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444119:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000150f4157[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444121:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001a4ab91f 17ffffc0001021 2:0 ffff9d819011aee0 78 lru 00000080:00200000:2.0:1775764923.444124:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006e0b95fa 78: 1< 0 - - > 2< 319488 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444125:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000150f4157 00000080:00200000:2.0:1775764923.444126:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 79. 00000080:00200000:2.0:1775764923.444127:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 79 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444134:0:34105:0:(rw26.c:858:ll_write_end()) pos 323584, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444136:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fb640a7d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444138:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000035eb2ef 17ffffc0001001 3:0 ffff9d819011ad20 79 lru 00000080:00200000:2.0:1775764923.444141:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003e62d90e 79: 1< 0 - - > 2< 323584 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444142:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fb640a7d 00000080:00200000:2.0:1775764923.444143:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 80. 00000080:00200000:2.0:1775764923.444144:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 80 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444150:0:34105:0:(rw26.c:858:ll_write_end()) pos 327680, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444152:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000081a25e1e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444153:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000553d8760 17ffffc0001001 3:0 ffff9d819011a7e0 80 lru 00000080:00200000:2.0:1775764923.444156:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000076d8021a 80: 1< 0 - - > 2< 327680 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444158:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000081a25e1e 00000080:00200000:2.0:1775764923.444158:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 81. 00000080:00200000:2.0:1775764923.444159:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 81 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444165:0:34105:0:(rw26.c:858:ll_write_end()) pos 331776, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444167:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001eb86f67[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444179:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f5211791 17ffffc0001001 3:0 ffff9d819011a8c0 81 lru 00000080:00200000:2.0:1775764923.444183:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f397583d 81: 1< 0 - - > 2< 331776 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444184:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001eb86f67 00000080:00200000:2.0:1775764923.444184:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 82. 00000080:00200000:2.0:1775764923.444186:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 82 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444194:0:34105:0:(rw26.c:858:ll_write_end()) pos 335872, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444196:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000044a171b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444198:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fefcffda 17ffffc0001001 3:0 ffff9d819011ac40 82 lru 00000080:00200000:2.0:1775764923.444201:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000523ef87b 82: 1< 0 - - > 2< 335872 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444202:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000044a171b 00000080:00200000:2.0:1775764923.444203:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 83. 00000080:00200000:2.0:1775764923.444204:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 83 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444210:0:34105:0:(rw26.c:858:ll_write_end()) pos 339968, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444212:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000bf9ec7e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444213:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000fa5c18a 17ffffc0001001 3:0 ffff9d819011aa80 83 lru 00000080:00200000:2.0:1775764923.444216:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cbb14cbd 83: 1< 0 - - > 2< 339968 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444218:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000bf9ec7e 00000080:00200000:2.0:1775764923.444218:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 84. 00000080:00200000:2.0:1775764923.444219:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 84 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444226:0:34105:0:(rw26.c:858:ll_write_end()) pos 344064, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444228:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000047e58426[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444229:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006e302466 17ffffc0001001 3:0 ffff9d819011a380 84 lru 00000080:00200000:2.0:1775764923.444232:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c610d168 84: 1< 0 - - > 2< 344064 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444233:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000047e58426 00000080:00200000:2.0:1775764923.444233:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 85. 00000080:00200000:2.0:1775764923.444234:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 85 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444241:0:34105:0:(rw26.c:858:ll_write_end()) pos 348160, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444243:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007a6f7002[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444244:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000048819d1d 17ffffc0001001 3:0 ffff9d819011ae00 85 lru 00000080:00200000:2.0:1775764923.444247:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000866569f6 85: 1< 0 - - > 2< 348160 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444248:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007a6f7002 00000080:00200000:2.0:1775764923.444249:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 86. 00000080:00200000:2.0:1775764923.444250:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 86 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444262:0:34105:0:(rw26.c:858:ll_write_end()) pos 352256, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444264:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003e9f1be8[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444266:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002d8505a7 17ffffc0001001 3:0 ffff9d819843a9a0 86 lru 00000080:00200000:2.0:1775764923.444269:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006970da51 86: 1< 0 - - > 2< 352256 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444270:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003e9f1be8 00000080:00200000:2.0:1775764923.444270:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 87. 00000080:00200000:2.0:1775764923.444271:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 87 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444278:0:34105:0:(rw26.c:858:ll_write_end()) pos 356352, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444280:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008caa44fa[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444281:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b3ecefc5 17ffffc0001001 3:0 ffff9d819843a540 87 lru 00000080:00200000:2.0:1775764923.444284:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000088389f0a 87: 1< 0 - - > 2< 356352 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444286:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008caa44fa 00000080:00200000:2.0:1775764923.444287:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 88. 00000080:00200000:2.0:1775764923.444287:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 88 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444294:0:34105:0:(rw26.c:858:ll_write_end()) pos 360448, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444296:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000020dd2f8c[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444297:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000672056f8 17ffffc0001001 3:0 ffff9d819843a000 88 lru 00000080:00200000:2.0:1775764923.444300:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000066fe7930 88: 1< 0 - - > 2< 360448 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444302:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000020dd2f8c 00000080:00200000:2.0:1775764923.444302:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 89. 00000080:00200000:2.0:1775764923.444303:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 89 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444310:0:34105:0:(rw26.c:858:ll_write_end()) pos 364544, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444312:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009c3fcd50[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444313:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fb766ad7 17ffffc0001001 3:0 ffff9d819843a2a0 89 lru 00000080:00200000:2.0:1775764923.444316:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a78d946e 89: 1< 0 - - > 2< 364544 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444317:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009c3fcd50 00000080:00200000:2.0:1775764923.444318:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 90. 00000080:00200000:2.0:1775764923.444319:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 90 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444325:0:34105:0:(rw26.c:858:ll_write_end()) pos 368640, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444327:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009ffe61ec[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444329:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004676d243 17ffffc0001001 3:0 ffff9d819843aee0 90 lru 00000080:00200000:2.0:1775764923.444332:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002e697661 90: 1< 0 - - > 2< 368640 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444333:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009ffe61ec 00000080:00200000:2.0:1775764923.444333:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 91. 00000080:00200000:2.0:1775764923.444334:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 91 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444340:0:34105:0:(rw26.c:858:ll_write_end()) pos 372736, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444342:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000ab48e45[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444343:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001930f922 17ffffc0001001 3:0 ffff9d819843ad20 91 lru 00000080:00200000:2.0:1775764923.444346:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003afcfb40 91: 1< 0 - - > 2< 372736 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444347:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000ab48e45 00000080:00200000:2.0:1775764923.444348:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 92. 00000080:00200000:2.0:1775764923.444349:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 92 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444355:0:34105:0:(rw26.c:858:ll_write_end()) pos 376832, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444357:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b7823d99[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444359:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f21a82c2 17ffffc0001001 3:0 ffff9d819843a7e0 92 lru 00000080:00200000:2.0:1775764923.444362:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006423c01a 92: 1< 0 - - > 2< 376832 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444363:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b7823d99 00000080:00200000:2.0:1775764923.444364:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 93. 00000080:00200000:2.0:1775764923.444365:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 93 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444374:0:34105:0:(rw26.c:858:ll_write_end()) pos 380928, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444376:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007e0627f4[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444377:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000adc10f5 17ffffc0001021 2:0 ffff9d819843a8c0 93 lru 00000080:00200000:2.0:1775764923.444381:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006c46696d 93: 1< 0 - - > 2< 380928 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444382:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007e0627f4 00000080:00200000:2.0:1775764923.444383:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 94. 00000080:00200000:2.0:1775764923.444383:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 94 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444390:0:34105:0:(rw26.c:858:ll_write_end()) pos 385024, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444393:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004ca23d92[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444394:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000069eaf104 17ffffc0001001 3:0 ffff9d819843ac40 94 lru 00000080:00200000:2.0:1775764923.444397:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000035f2a883 94: 1< 0 - - > 2< 385024 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444398:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004ca23d92 00000080:00200000:2.0:1775764923.444399:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 95. 00000080:00200000:2.0:1775764923.444400:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 95 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444406:0:34105:0:(rw26.c:858:ll_write_end()) pos 389120, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444408:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000051820974[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444409:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000101285d6 17ffffc0001001 3:0 ffff9d819843aa80 95 lru 00000080:00200000:2.0:1775764923.444412:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000634d654b 95: 1< 0 - - > 2< 389120 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444413:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000051820974 00000080:00200000:2.0:1775764923.444414:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 96. 00000080:00200000:2.0:1775764923.444415:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 96 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444421:0:34105:0:(rw26.c:858:ll_write_end()) pos 393216, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444423:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000020a939f5[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444424:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001294d333 17ffffc0001001 3:0 ffff9d819843a380 96 lru 00000080:00200000:2.0:1775764923.444427:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cb9fe8f3 96: 1< 0 - - > 2< 393216 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444428:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000020a939f5 00000080:00200000:2.0:1775764923.444429:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 97. 00000080:00200000:2.0:1775764923.444430:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 97 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444436:0:34105:0:(rw26.c:858:ll_write_end()) pos 397312, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444438:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000744d5ab8[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444439:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b892948d 17ffffc0001001 3:0 ffff9d819843ae00 97 lru 00000080:00200000:2.0:1775764923.444442:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ad4d730e 97: 1< 0 - - > 2< 397312 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444443:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000744d5ab8 00000080:00200000:2.0:1775764923.444444:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 98. 00000080:00200000:2.0:1775764923.444444:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 98 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444451:0:34105:0:(rw26.c:858:ll_write_end()) pos 401408, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444452:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d896bb37[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444454:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006a0c9152 17ffffc0001001 3:0 ffff9d819843a1c0 98 lru 00000080:00200000:2.0:1775764923.444457:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003f560af4 98: 1< 0 - - > 2< 401408 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444458:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d896bb37 00000080:00200000:2.0:1775764923.444458:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 99. 00000080:00200000:2.0:1775764923.444459:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 99 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444466:0:34105:0:(rw26.c:858:ll_write_end()) pos 405504, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444467:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b4a21eeb[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444469:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005d097571 17ffffc0001001 3:0 ffff9d819843a700 99 lru 00000080:00200000:2.0:1775764923.444472:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d369c337 99: 1< 0 - - > 2< 405504 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444474:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b4a21eeb 00000080:00200000:2.0:1775764923.444475:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 100. 00000080:00200000:2.0:1775764923.444477:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 100 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444489:0:34105:0:(rw26.c:858:ll_write_end()) pos 409600, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444493:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000946e3540[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444495:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000048f0ae69 17ffffc0001001 3:0 ffff9d819843a620 100 lru 00000080:00200000:2.0:1775764923.444501:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000235243d9 100: 1< 0 - - > 2< 409600 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444503:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000946e3540 00000080:00200000:2.0:1775764923.444504:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 101. 00000080:00200000:2.0:1775764923.444505:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 101 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444517:0:34105:0:(rw26.c:858:ll_write_end()) pos 413696, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444521:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f33909f0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444524:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000045a5db6 17ffffc0001001 3:0 ffff9d819843a460 101 lru 00000080:00200000:2.0:1775764923.444527:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000aea39203 101: 1< 0 - - > 2< 413696 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444529:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f33909f0 00000080:00200000:2.0:1775764923.444529:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 102. 00000080:00200000:2.0:1775764923.444531:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 102 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444539:0:34105:0:(rw26.c:858:ll_write_end()) pos 417792, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444541:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002165ab30[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444543:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f27a5b7e 17ffffc0001001 3:0 ffff9d819843a0e0 102 lru 00000080:00200000:2.0:1775764923.444546:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fc7cf3d5 102: 1< 0 - - > 2< 417792 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444547:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002165ab30 00000080:00200000:2.0:1775764923.444548:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 103. 00000080:00200000:2.0:1775764923.444548:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 103 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444557:0:34105:0:(rw26.c:858:ll_write_end()) pos 421888, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444561:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ff0b62c7[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444563:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000733f6b6c 17ffffc0001001 3:0 ffff9d819843ab60 103 lru 00000080:00200000:2.0:1775764923.444568:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000eadf743c 103: 1< 0 - - > 2< 421888 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444570:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ff0b62c7 00000080:00200000:2.0:1775764923.444571:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 104. 00000080:00200000:2.0:1775764923.444573:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 104 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444592:0:34105:0:(rw26.c:858:ll_write_end()) pos 425984, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444596:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a2a0332c[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444599:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000dab1e56f 17ffffc0001001 3:0 ffff9d81870a77e0 104 lru 00000080:00200000:2.0:1775764923.444604:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007ff8990f 104: 1< 0 - - > 2< 425984 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444606:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a2a0332c 00000080:00200000:2.0:1775764923.444607:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 105. 00000080:00200000:2.0:1775764923.444609:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 105 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444619:0:34105:0:(rw26.c:858:ll_write_end()) pos 430080, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444621:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fe31c888[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444623:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d16cb87a 17ffffc0001001 3:0 ffff9d81870a78c0 105 lru 00000080:00200000:2.0:1775764923.444626:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cf6905eb 105: 1< 0 - - > 2< 430080 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444628:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fe31c888 00000080:00200000:2.0:1775764923.444628:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 106. 00000080:00200000:2.0:1775764923.444629:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 106 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444636:0:34105:0:(rw26.c:858:ll_write_end()) pos 434176, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444638:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009ad1d74f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444639:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000005e81d7a 17ffffc0001001 3:0 ffff9d81870a7c40 106 lru 00000080:00200000:2.0:1775764923.444642:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000025221e6 106: 1< 0 - - > 2< 434176 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444643:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009ad1d74f 00000080:00200000:2.0:1775764923.444644:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 107. 00000080:00200000:2.0:1775764923.444645:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 107 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444652:0:34105:0:(rw26.c:858:ll_write_end()) pos 438272, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444654:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004ea06b56[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444656:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002650a203 17ffffc0001001 3:0 ffff9d81870a7a80 107 lru 00000080:00200000:2.0:1775764923.444659:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000010f51677 107: 1< 0 - - > 2< 438272 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444660:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004ea06b56 00000080:00200000:2.0:1775764923.444661:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 108. 00000080:00200000:2.0:1775764923.444662:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 108 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444670:0:34105:0:(rw26.c:858:ll_write_end()) pos 442368, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444672:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005608ae3b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444674:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006a5bde6d 17ffffc0001021 2:0 ffff9d81870a7380 108 lru 00000080:00200000:2.0:1775764923.444677:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f4ab334d 108: 1< 0 - - > 2< 442368 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444678:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005608ae3b 00000080:00200000:2.0:1775764923.444679:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 109. 00000080:00200000:2.0:1775764923.444680:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 109 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444686:0:34105:0:(rw26.c:858:ll_write_end()) pos 446464, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444688:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cd238dd6[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444690:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006e6d3d21 17ffffc0001001 3:0 ffff9d81870a7e00 109 lru 00000080:00200000:2.0:1775764923.444693:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008a95cf69 109: 1< 0 - - > 2< 446464 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444694:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cd238dd6 00000080:00200000:2.0:1775764923.444694:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 110. 00000080:00200000:2.0:1775764923.444695:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 110 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444703:0:34105:0:(rw26.c:858:ll_write_end()) pos 450560, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444704:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000240aca2b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444706:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006386179f 17ffffc0001001 3:0 ffff9d81870a71c0 110 lru 00000080:00200000:2.0:1775764923.444709:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003313839e 110: 1< 0 - - > 2< 450560 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444710:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000240aca2b 00000080:00200000:2.0:1775764923.444711:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 111. 00000080:00200000:2.0:1775764923.444712:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 111 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444718:0:34105:0:(rw26.c:858:ll_write_end()) pos 454656, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444720:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009c2cfbaf[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444721:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c1b2bde5 17ffffc0001001 3:0 ffff9d81870a7700 111 lru 00000080:00200000:2.0:1775764923.444724:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bf273f51 111: 1< 0 - - > 2< 454656 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444725:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009c2cfbaf 00000080:00200000:2.0:1775764923.444726:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 112. 00000080:00200000:2.0:1775764923.444727:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 112 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444734:0:34105:0:(rw26.c:858:ll_write_end()) pos 458752, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444736:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d7a22b29[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444738:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e37d893f 17ffffc0001001 3:0 ffff9d81870a7620 112 lru 00000080:00200000:2.0:1775764923.444741:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000092880810 112: 1< 0 - - > 2< 458752 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444742:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d7a22b29 00000080:00200000:2.0:1775764923.444742:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 113. 00000080:00200000:2.0:1775764923.444743:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 113 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444749:0:34105:0:(rw26.c:858:ll_write_end()) pos 462848, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444751:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000eaa680dd[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444753:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000028d2eb14 17ffffc0001001 3:0 ffff9d81870a7460 113 lru 00000080:00200000:2.0:1775764923.444756:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000037417775 113: 1< 0 - - > 2< 462848 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444757:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000eaa680dd 00000080:00200000:2.0:1775764923.444757:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 114. 00000080:00200000:2.0:1775764923.444758:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 114 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444765:0:34105:0:(rw26.c:858:ll_write_end()) pos 466944, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444767:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e9545b70[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444769:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000782d2816 17ffffc0001001 3:0 ffff9d81870a70e0 114 lru 00000080:00200000:2.0:1775764923.444772:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000f2cbf38 114: 1< 0 - - > 2< 466944 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444773:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e9545b70 00000080:00200000:2.0:1775764923.444773:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 115. 00000080:00200000:2.0:1775764923.444774:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 115 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444781:0:34105:0:(rw26.c:858:ll_write_end()) pos 471040, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444783:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000f570de2[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444784:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005a52053c 17ffffc0001001 3:0 ffff9d81870a7b60 115 lru 00000080:00200000:2.0:1775764923.444787:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000859e6de1 115: 1< 0 - - > 2< 471040 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444788:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000f570de2 00000080:00200000:2.0:1775764923.444789:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 116. 00000080:00200000:2.0:1775764923.444789:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 116 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444796:0:34105:0:(rw26.c:858:ll_write_end()) pos 475136, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444797:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006b1e08d8[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444799:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001b2a571f 17ffffc0001001 3:0 ffff9d81870a79a0 116 lru 00000080:00200000:2.0:1775764923.444802:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000045d229ea 116: 1< 0 - - > 2< 475136 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444803:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006b1e08d8 00000080:00200000:2.0:1775764923.444804:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 117. 00000080:00200000:2.0:1775764923.444804:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 117 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444811:0:34105:0:(rw26.c:858:ll_write_end()) pos 479232, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444813:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000034d38f23[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444814:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006d7d0287 17ffffc0001001 3:0 ffff9d81870a7540 117 lru 00000080:00200000:2.0:1775764923.444817:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009e8662ef 117: 1< 0 - - > 2< 479232 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444818:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000034d38f23 00000080:00200000:2.0:1775764923.444818:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 118. 00000080:00200000:2.0:1775764923.444819:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 118 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444826:0:34105:0:(rw26.c:858:ll_write_end()) pos 483328, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444828:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000d6b9a4b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444829:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000dfae8dad 17ffffc0001001 3:0 ffff9d81870a7000 118 lru 00000080:00200000:2.0:1775764923.444832:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001be3ac54 118: 1< 0 - - > 2< 483328 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444833:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000d6b9a4b 00000080:00200000:2.0:1775764923.444834:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 119. 00000080:00200000:2.0:1775764923.444835:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 119 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444841:0:34105:0:(rw26.c:858:ll_write_end()) pos 487424, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444843:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000097552c7b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444845:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c3d16b6d 17ffffc0001001 3:0 ffff9d81870a72a0 119 lru 00000080:00200000:2.0:1775764923.444847:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000005def38d 119: 1< 0 - - > 2< 487424 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444848:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000097552c7b 00000080:00200000:2.0:1775764923.444849:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 120. 00000080:00200000:2.0:1775764923.444850:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 120 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444856:0:34105:0:(rw26.c:858:ll_write_end()) pos 491520, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444858:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002a6ae905[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444859:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cdd55a33 17ffffc0001001 3:0 ffff9d81870a7ee0 120 lru 00000080:00200000:2.0:1775764923.444862:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000098241dee 120: 1< 0 - - > 2< 491520 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444864:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002a6ae905 00000080:00200000:2.0:1775764923.444864:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 121. 00000080:00200000:2.0:1775764923.444865:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 121 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444906:0:34105:0:(rw26.c:858:ll_write_end()) pos 495616, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444911:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f04e2180[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444913:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d58f975b 17ffffc0001001 3:0 ffff9d81870a7d20 121 lru 00000080:00200000:2.0:1775764923.444918:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cf892736 121: 1< 0 - - > 2< 495616 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444921:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f04e2180 00000080:00200000:2.0:1775764923.444922:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 122. 00000080:00200000:2.0:1775764923.444924:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 122 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444943:0:34105:0:(rw26.c:858:ll_write_end()) pos 499712, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444947:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000edc57e6e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444949:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003e36b825 17ffffc0001001 3:0 ffff9d8198126460 122 lru 00000080:00200000:2.0:1775764923.444952:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008d31a32d 122: 1< 0 - - > 2< 499712 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444954:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000edc57e6e 00000080:00200000:2.0:1775764923.444954:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 123. 00000080:00200000:2.0:1775764923.444955:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 123 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444966:0:34105:0:(rw26.c:858:ll_write_end()) pos 503808, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444968:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000969c9c79[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444970:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f1ec637e 17ffffc0001021 2:0 ffff9d81981260e0 123 lru 00000080:00200000:2.0:1775764923.444973:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000095db1375 123: 1< 0 - - > 2< 503808 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.444975:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000969c9c79 00000080:00200000:2.0:1775764923.444976:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 124. 00000080:00200000:2.0:1775764923.444977:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 124 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.444987:0:34105:0:(rw26.c:858:ll_write_end()) pos 507904, len 4096, copied 4096 00000080:00200000:2.0:1775764923.444990:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000f726771[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.444993:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000499e600e 17ffffc0001001 3:0 ffff9d8198126b60 124 lru 00000080:00200000:2.0:1775764923.444998:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003c394713 124: 1< 0 - - > 2< 507904 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445000:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000f726771 00000080:00200000:2.0:1775764923.445001:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 125. 00000080:00200000:2.0:1775764923.445003:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 125 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445015:0:34105:0:(rw26.c:858:ll_write_end()) pos 512000, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445019:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000051c52ed3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445021:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000064afc3f8 17ffffc0001001 3:0 ffff9d81981269a0 125 lru 00000080:00200000:2.0:1775764923.445026:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003f592b55 125: 1< 0 - - > 2< 512000 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445028:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000051c52ed3 00000080:00200000:2.0:1775764923.445029:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 126. 00000080:00200000:2.0:1775764923.445030:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 126 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445041:0:34105:0:(rw26.c:858:ll_write_end()) pos 516096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445044:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007ecf116d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445047:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005742ab95 17ffffc0001001 3:0 ffff9d8198126540 126 lru 00000080:00200000:2.0:1775764923.445052:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004cd85c8b 126: 1< 0 - - > 2< 516096 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445054:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007ecf116d 00000080:00200000:2.0:1775764923.445055:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 127. 00000080:00200000:2.0:1775764923.445057:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 127 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445067:0:34105:0:(rw26.c:858:ll_write_end()) pos 520192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445070:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009241345f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445071:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005274f1da 17ffffc0001001 3:0 ffff9d8198126000 127 lru 00000080:00200000:2.0:1775764923.445074:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ea88d06e 127: 1< 0 - - > 2< 520192 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445076:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009241345f 00000080:00200000:2.0:1775764923.445076:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 128. 00000080:00200000:2.0:1775764923.445077:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 128 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445086:0:34105:0:(rw26.c:858:ll_write_end()) pos 524288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445089:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000aa3f9829[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445090:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ce2a0251 17ffffc0001001 3:0 ffff9d81981262a0 128 lru 00000080:00200000:2.0:1775764923.445093:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000919f435d 128: 1< 0 - - > 2< 524288 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445095:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000aa3f9829 00000080:00200000:2.0:1775764923.445095:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 129. 00000080:00200000:2.0:1775764923.445096:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 129 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445103:0:34105:0:(rw26.c:858:ll_write_end()) pos 528384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445105:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000299e52b3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445106:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000780b9045 17ffffc0001001 3:0 ffff9d8198126ee0 129 lru 00000080:00200000:2.0:1775764923.445110:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b9b95dc4 129: 1< 0 - - > 2< 528384 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445111:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000299e52b3 00000080:00200000:2.0:1775764923.445112:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 130. 00000080:00200000:2.0:1775764923.445112:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 130 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445119:0:34105:0:(rw26.c:858:ll_write_end()) pos 532480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445121:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000f2453a9[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445123:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000099e0fcc3 17ffffc0001001 3:0 ffff9d8198126d20 130 lru 00000080:00200000:2.0:1775764923.445126:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000205a7075 130: 1< 0 - - > 2< 532480 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445127:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000f2453a9 00000080:00200000:2.0:1775764923.445127:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 131. 00000080:00200000:2.0:1775764923.445128:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 131 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445134:0:34105:0:(rw26.c:858:ll_write_end()) pos 536576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445136:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d7c9bcbd[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445138:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000010600f57 17ffffc0001001 3:0 ffff9d81981267e0 131 lru 00000080:00200000:2.0:1775764923.445141:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000df9cb361 131: 1< 0 - - > 2< 536576 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445142:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d7c9bcbd 00000080:00200000:2.0:1775764923.445143:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 132. 00000080:00200000:2.0:1775764923.445144:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 132 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445150:0:34105:0:(rw26.c:858:ll_write_end()) pos 540672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445152:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e0fb51f7[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445153:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003f4235e2 17ffffc0001001 3:0 ffff9d81981268c0 132 lru 00000080:00200000:2.0:1775764923.445156:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d9494a4c 132: 1< 0 - - > 2< 540672 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445157:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e0fb51f7 00000080:00200000:2.0:1775764923.445158:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 133. 00000080:00200000:2.0:1775764923.445158:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 133 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445165:0:34105:0:(rw26.c:858:ll_write_end()) pos 544768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445166:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b3c23d3b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445168:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000fd207e3b 17ffffc0001001 3:0 ffff9d8198126c40 133 lru 00000080:00200000:2.0:1775764923.445186:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007765491c 133: 1< 0 - - > 2< 544768 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445187:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b3c23d3b 00000080:00200000:2.0:1775764923.445187:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 134. 00000080:00200000:2.0:1775764923.445188:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 134 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445198:0:34105:0:(rw26.c:858:ll_write_end()) pos 548864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445200:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e39f3554[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445201:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000891f96aa 17ffffc0001001 3:0 ffff9d8198126a80 134 lru 00000080:00200000:2.0:1775764923.445204:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ef9c9d96 134: 1< 0 - - > 2< 548864 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445206:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e39f3554 00000080:00200000:2.0:1775764923.445207:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 135. 00000080:00200000:2.0:1775764923.445207:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 135 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445215:0:34105:0:(rw26.c:858:ll_write_end()) pos 552960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445217:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000de753469[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445219:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000078dc0f46 17ffffc0001001 3:0 ffff9d8198126380 135 lru 00000080:00200000:2.0:1775764923.445222:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000053570dec 135: 1< 0 - - > 2< 552960 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445223:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000de753469 00000080:00200000:2.0:1775764923.445224:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 136. 00000080:00200000:2.0:1775764923.445224:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 136 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445231:0:34105:0:(rw26.c:858:ll_write_end()) pos 557056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445233:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008f7f17e0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445234:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000024e225ec 17ffffc0001001 3:0 ffff9d8198126e00 136 lru 00000080:00200000:2.0:1775764923.445237:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005deb63cf 136: 1< 0 - - > 2< 557056 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445238:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008f7f17e0 00000080:00200000:2.0:1775764923.445239:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 137. 00000080:00200000:2.0:1775764923.445239:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 137 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445246:0:34105:0:(rw26.c:858:ll_write_end()) pos 561152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445248:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ea19cfcf[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445249:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a11febfa 17ffffc0001001 3:0 ffff9d81981261c0 137 lru 00000080:00200000:2.0:1775764923.445252:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008c558033 137: 1< 0 - - > 2< 561152 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445253:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ea19cfcf 00000080:00200000:2.0:1775764923.445254:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 138. 00000080:00200000:2.0:1775764923.445254:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 138 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445263:0:34105:0:(rw26.c:858:ll_write_end()) pos 565248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445265:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005fa0b8b0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445267:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000016f89fe2 17ffffc0001021 2:0 ffff9d8198126700 138 lru 00000080:00200000:2.0:1775764923.445270:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000036be1ba5 138: 1< 0 - - > 2< 565248 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445271:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005fa0b8b0 00000080:00200000:2.0:1775764923.445271:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 139. 00000080:00200000:2.0:1775764923.445272:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 139 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445279:0:34105:0:(rw26.c:858:ll_write_end()) pos 569344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445281:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000069c65556[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445282:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000180020bb 17ffffc0001001 3:0 ffff9d8198126620 139 lru 00000080:00200000:2.0:1775764923.445285:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000074ffe496 139: 1< 0 - - > 2< 569344 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445286:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000069c65556 00000080:00200000:2.0:1775764923.445287:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 140. 00000080:00200000:2.0:1775764923.445288:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 140 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445299:0:34105:0:(rw26.c:858:ll_write_end()) pos 573440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445301:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007667428f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445303:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e7a049eb 17ffffc0001001 3:0 ffff9d8185814000 140 lru 00000080:00200000:2.0:1775764923.445306:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008a7cc3f8 140: 1< 0 - - > 2< 573440 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445307:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007667428f 00000080:00200000:2.0:1775764923.445307:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 141. 00000080:00200000:2.0:1775764923.445308:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 141 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445315:0:34105:0:(rw26.c:858:ll_write_end()) pos 577536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445317:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000039090f9a[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445319:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000043ef6b00 17ffffc0001001 3:0 ffff9d81858142a0 141 lru 00000080:00200000:2.0:1775764923.445322:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000747b5bd7 141: 1< 0 - - > 2< 577536 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445323:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000039090f9a 00000080:00200000:2.0:1775764923.445324:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 142. 00000080:00200000:2.0:1775764923.445324:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 142 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445331:0:34105:0:(rw26.c:858:ll_write_end()) pos 581632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445333:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a826796b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445335:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000028ce0f95 17ffffc0001001 3:0 ffff9d8185814ee0 142 lru 00000080:00200000:2.0:1775764923.445337:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000009a5c5f8 142: 1< 0 - - > 2< 581632 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445339:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a826796b 00000080:00200000:2.0:1775764923.445339:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 143. 00000080:00200000:2.0:1775764923.445340:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 143 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445346:0:34105:0:(rw26.c:858:ll_write_end()) pos 585728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445349:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d255fc75[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445351:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d049111f 17ffffc0001001 3:0 ffff9d8185814d20 143 lru 00000080:00200000:2.0:1775764923.445356:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000013405dfe 143: 1< 0 - - > 2< 585728 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445358:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d255fc75 00000080:00200000:2.0:1775764923.445359:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 144. 00000080:00200000:2.0:1775764923.445361:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 144 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445373:0:34105:0:(rw26.c:858:ll_write_end()) pos 589824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445376:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006d46e519[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445379:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c8400411 17ffffc0001001 3:0 ffff9d81858147e0 144 lru 00000080:00200000:2.0:1775764923.445384:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000de53c12a 144: 1< 0 - - > 2< 589824 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445386:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006d46e519 00000080:00200000:2.0:1775764923.445387:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 145. 00000080:00200000:2.0:1775764923.445389:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 145 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445401:0:34105:0:(rw26.c:858:ll_write_end()) pos 593920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445405:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cd3478b9[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445408:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c230e698 17ffffc0001001 3:0 ffff9d81858148c0 145 lru 00000080:00200000:2.0:1775764923.445413:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000077e8a22 145: 1< 0 - - > 2< 593920 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445415:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cd3478b9 00000080:00200000:2.0:1775764923.445415:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 146. 00000080:00200000:2.0:1775764923.445417:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 146 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445426:0:34105:0:(rw26.c:858:ll_write_end()) pos 598016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445428:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bad7dd07[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445430:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001bf1fd20 17ffffc0001001 3:0 ffff9d8185814c40 146 lru 00000080:00200000:2.0:1775764923.445433:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006b32a00b 146: 1< 0 - - > 2< 598016 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445434:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bad7dd07 00000080:00200000:2.0:1775764923.445435:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 147. 00000080:00200000:2.0:1775764923.445436:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 147 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445443:0:34105:0:(rw26.c:858:ll_write_end()) pos 602112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445444:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d6ab7dc4[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445446:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002cf01dd2 17ffffc0001001 3:0 ffff9d8185814a80 147 lru 00000080:00200000:2.0:1775764923.445449:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f8673768 147: 1< 0 - - > 2< 602112 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445450:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d6ab7dc4 00000080:00200000:2.0:1775764923.445451:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 148. 00000080:00200000:2.0:1775764923.445452:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 148 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445458:0:34105:0:(rw26.c:858:ll_write_end()) pos 606208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445460:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000660d2a4f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445462:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b6ebb9c3 17ffffc0001001 3:0 ffff9d8185814380 148 lru 00000080:00200000:2.0:1775764923.445465:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005ad634f0 148: 1< 0 - - > 2< 606208 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445466:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000660d2a4f 00000080:00200000:2.0:1775764923.445467:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 149. 00000080:00200000:2.0:1775764923.445468:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 149 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445478:0:34105:0:(rw26.c:858:ll_write_end()) pos 610304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445482:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000069e454b2[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445484:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000029b9cf74 17ffffc0001001 3:0 ffff9d8185814e00 149 lru 00000080:00200000:2.0:1775764923.445489:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009cbdb3db 149: 1< 0 - - > 2< 610304 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445491:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000069e454b2 00000080:00200000:2.0:1775764923.445492:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 150. 00000080:00200000:2.0:1775764923.445493:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 150 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445505:0:34105:0:(rw26.c:858:ll_write_end()) pos 614400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445508:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ee6493dc[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445511:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000072689822 17ffffc0001001 3:0 ffff9d81858141c0 150 lru 00000080:00200000:2.0:1775764923.445516:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c5a3cac9 150: 1< 0 - - > 2< 614400 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445518:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ee6493dc 00000080:00200000:2.0:1775764923.445518:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 151. 00000080:00200000:2.0:1775764923.445520:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 151 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445529:0:34105:0:(rw26.c:858:ll_write_end()) pos 618496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445531:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000056869d94[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445532:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d597805e 17ffffc0001001 3:0 ffff9d8185814700 151 lru 00000080:00200000:2.0:1775764923.445536:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000180d6283 151: 1< 0 - - > 2< 618496 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445537:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000056869d94 00000080:00200000:2.0:1775764923.445537:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 152. 00000080:00200000:2.0:1775764923.445538:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 152 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445545:0:34105:0:(rw26.c:858:ll_write_end()) pos 622592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445547:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000062e39bbe[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445548:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000026b89cdb 17ffffc0001001 3:0 ffff9d8185814620 152 lru 00000080:00200000:2.0:1775764923.445551:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000524aa6e9 152: 1< 0 - - > 2< 622592 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445553:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000062e39bbe 00000080:00200000:2.0:1775764923.445553:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 153. 00000080:00200000:2.0:1775764923.445554:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 153 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445563:0:34105:0:(rw26.c:858:ll_write_end()) pos 626688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445565:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006ba1dc00[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445567:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008da06a81 17ffffc0001021 2:0 ffff9d8185814460 153 lru 00000080:00200000:2.0:1775764923.445570:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ac451fd5 153: 1< 0 - - > 2< 626688 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445571:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006ba1dc00 00000080:00200000:2.0:1775764923.445572:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 154. 00000080:00200000:2.0:1775764923.445572:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 154 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445579:0:34105:0:(rw26.c:858:ll_write_end()) pos 630784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445581:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e5aeba95[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445582:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006fa4929a 17ffffc0001001 3:0 ffff9d81858140e0 154 lru 00000080:00200000:2.0:1775764923.445585:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000db956106 154: 1< 0 - - > 2< 630784 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445586:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e5aeba95 00000080:00200000:2.0:1775764923.445587:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 155. 00000080:00200000:2.0:1775764923.445588:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 155 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445594:0:34105:0:(rw26.c:858:ll_write_end()) pos 634880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445596:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ad3842c0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445597:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003130ff08 17ffffc0001001 3:0 ffff9d8185814b60 155 lru 00000080:00200000:2.0:1775764923.445600:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c0fa363f 155: 1< 0 - - > 2< 634880 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445601:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ad3842c0 00000080:00200000:2.0:1775764923.445602:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 156. 00000080:00200000:2.0:1775764923.445603:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 156 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445610:0:34105:0:(rw26.c:858:ll_write_end()) pos 638976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445612:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000c44fcf0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445613:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000043b02253 17ffffc0001001 3:0 ffff9d81858149a0 156 lru 00000080:00200000:2.0:1775764923.445616:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000037020c8f 156: 1< 0 - - > 2< 638976 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445617:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000c44fcf0 00000080:00200000:2.0:1775764923.445617:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 157. 00000080:00200000:2.0:1775764923.445618:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 157 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445625:0:34105:0:(rw26.c:858:ll_write_end()) pos 643072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445627:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000073f012ad[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445628:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000b2b2a6d 17ffffc0001001 3:0 ffff9d8185814540 157 lru 00000080:00200000:2.0:1775764923.445631:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002503b803 157: 1< 0 - - > 2< 643072 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445632:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000073f012ad 00000080:00200000:2.0:1775764923.445633:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 158. 00000080:00200000:2.0:1775764923.445633:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 158 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445647:0:34105:0:(rw26.c:858:ll_write_end()) pos 647168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445651:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002ea54806[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445653:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000007e0ffd4 17ffffc0001001 3:0 ffff9d8190f4f540 158 lru 00000080:00200000:2.0:1775764923.445658:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001fc36441 158: 1< 0 - - > 2< 647168 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445659:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002ea54806 00000080:00200000:2.0:1775764923.445660:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 159. 00000080:00200000:2.0:1775764923.445662:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 159 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445674:0:34105:0:(rw26.c:858:ll_write_end()) pos 651264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445677:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000239cf94e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445680:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007283925f 17ffffc0001001 3:0 ffff9d8190f4f000 159 lru 00000080:00200000:2.0:1775764923.445684:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000083af57b7 159: 1< 0 - - > 2< 651264 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445686:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000239cf94e 00000080:00200000:2.0:1775764923.445687:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 160. 00000080:00200000:2.0:1775764923.445689:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 160 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445702:0:34105:0:(rw26.c:858:ll_write_end()) pos 655360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445705:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a90b69a8[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445708:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a4076c10 17ffffc0001001 3:0 ffff9d8190f4f2a0 160 lru 00000080:00200000:2.0:1775764923.445713:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c23079f9 160: 1< 0 - - > 2< 655360 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445715:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a90b69a8 00000080:00200000:2.0:1775764923.445716:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 161. 00000080:00200000:2.0:1775764923.445717:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 161 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445728:0:34105:0:(rw26.c:858:ll_write_end()) pos 659456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445730:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005a4ff0b6[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445732:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000029f844d9 17ffffc0001001 3:0 ffff9d8190f4fee0 161 lru 00000080:00200000:2.0:1775764923.445735:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d91e654d 161: 1< 0 - - > 2< 659456 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445736:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005a4ff0b6 00000080:00200000:2.0:1775764923.445737:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 162. 00000080:00200000:2.0:1775764923.445737:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 162 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445744:0:34105:0:(rw26.c:858:ll_write_end()) pos 663552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445746:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005a3151af[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445747:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ecd8ccf8 17ffffc0001001 3:0 ffff9d8190f4fd20 162 lru 00000080:00200000:2.0:1775764923.445750:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000094cbbbb6 162: 1< 0 - - > 2< 663552 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445751:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005a3151af 00000080:00200000:2.0:1775764923.445752:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 163. 00000080:00200000:2.0:1775764923.445753:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 163 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445761:0:34105:0:(rw26.c:858:ll_write_end()) pos 667648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445763:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000231960e9[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445765:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000f447bd2 17ffffc0001001 3:0 ffff9d8190f4f7e0 163 lru 00000080:00200000:2.0:1775764923.445768:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001e90d201 163: 1< 0 - - > 2< 667648 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445769:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000231960e9 00000080:00200000:2.0:1775764923.445770:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 164. 00000080:00200000:2.0:1775764923.445770:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 164 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445777:0:34105:0:(rw26.c:858:ll_write_end()) pos 671744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445779:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000074d700f3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445780:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009c5cdbc3 17ffffc0001001 3:0 ffff9d8190f4f8c0 164 lru 00000080:00200000:2.0:1775764923.445783:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fc6d17aa 164: 1< 0 - - > 2< 671744 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445784:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000074d700f3 00000080:00200000:2.0:1775764923.445785:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 165. 00000080:00200000:2.0:1775764923.445786:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 165 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445792:0:34105:0:(rw26.c:858:ll_write_end()) pos 675840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445794:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a5715a40[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445795:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f7f61fbf 17ffffc0001001 3:0 ffff9d8190f4fc40 165 lru 00000080:00200000:2.0:1775764923.445798:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009d3c1221 165: 1< 0 - - > 2< 675840 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445799:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a5715a40 00000080:00200000:2.0:1775764923.445800:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 166. 00000080:00200000:2.0:1775764923.445800:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 166 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445807:0:34105:0:(rw26.c:858:ll_write_end()) pos 679936, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445809:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000039bc38f1[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445811:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000099fc1184 17ffffc0001001 3:0 ffff9d8190f4fa80 166 lru 00000080:00200000:2.0:1775764923.445814:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001b0e7e20 166: 1< 0 - - > 2< 679936 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445815:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000039bc38f1 00000080:00200000:2.0:1775764923.445815:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 167. 00000080:00200000:2.0:1775764923.445816:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 167 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445823:0:34105:0:(rw26.c:858:ll_write_end()) pos 684032, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445825:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009fe515fd[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445826:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001128c330 17ffffc0001001 3:0 ffff9d8190f4f380 167 lru 00000080:00200000:2.0:1775764923.445829:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000011b44b9e 167: 1< 0 - - > 2< 684032 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445830:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009fe515fd 00000080:00200000:2.0:1775764923.445831:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 168. 00000080:00200000:2.0:1775764923.445832:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 168 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445840:0:34105:0:(rw26.c:858:ll_write_end()) pos 688128, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445842:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006f0b00a4[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445844:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000492ded04 17ffffc0001021 2:0 ffff9d8190f4fe00 168 lru 00000080:00200000:2.0:1775764923.445847:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000056e68498 168: 1< 0 - - > 2< 688128 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445848:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006f0b00a4 00000080:00200000:2.0:1775764923.445849:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 169. 00000080:00200000:2.0:1775764923.445850:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 169 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445856:0:34105:0:(rw26.c:858:ll_write_end()) pos 692224, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445858:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006d5a03b1[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445860:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000874a6b6c 17ffffc0001001 3:0 ffff9d8190f4f1c0 169 lru 00000080:00200000:2.0:1775764923.445862:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dd6b96b0 169: 1< 0 - - > 2< 692224 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445864:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006d5a03b1 00000080:00200000:2.0:1775764923.445865:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 170. 00000080:00200000:2.0:1775764923.445866:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 170 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445873:0:34105:0:(rw26.c:858:ll_write_end()) pos 696320, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445887:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d357445d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445889:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ee5273fd 17ffffc0001001 3:0 ffff9d8190f4f700 170 lru 00000080:00200000:2.0:1775764923.445892:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d69f59d7 170: 1< 0 - - > 2< 696320 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445894:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d357445d 00000080:00200000:2.0:1775764923.445895:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 171. 00000080:00200000:2.0:1775764923.445896:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 171 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445904:0:34105:0:(rw26.c:858:ll_write_end()) pos 700416, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445907:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000663d1b9f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445908:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001c0b8588 17ffffc0001001 3:0 ffff9d8190f4f620 171 lru 00000080:00200000:2.0:1775764923.445911:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004aac8d23 171: 1< 0 - - > 2< 700416 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445913:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000663d1b9f 00000080:00200000:2.0:1775764923.445913:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 172. 00000080:00200000:2.0:1775764923.445914:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 172 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445921:0:34105:0:(rw26.c:858:ll_write_end()) pos 704512, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445923:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008967b74f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445924:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000abd65ee5 17ffffc0001001 3:0 ffff9d8190f4f460 172 lru 00000080:00200000:2.0:1775764923.445927:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dd8a5145 172: 1< 0 - - > 2< 704512 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445928:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008967b74f 00000080:00200000:2.0:1775764923.445929:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 173. 00000080:00200000:2.0:1775764923.445930:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 173 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445936:0:34105:0:(rw26.c:858:ll_write_end()) pos 708608, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445938:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fe8a8641[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445939:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000042dfef5f 17ffffc0001001 3:0 ffff9d8190f4f0e0 173 lru 00000080:00200000:2.0:1775764923.445942:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001ac5ef1c 173: 1< 0 - - > 2< 708608 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445944:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fe8a8641 00000080:00200000:2.0:1775764923.445944:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 174. 00000080:00200000:2.0:1775764923.445945:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 174 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445952:0:34105:0:(rw26.c:858:ll_write_end()) pos 712704, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445954:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000316d7965[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445955:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006922fb7b 17ffffc0001001 3:0 ffff9d8190f4fb60 174 lru 00000080:00200000:2.0:1775764923.445958:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e09892b1 174: 1< 0 - - > 2< 712704 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445959:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000316d7965 00000080:00200000:2.0:1775764923.445959:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 175. 00000080:00200000:2.0:1775764923.445960:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 175 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445967:0:34105:0:(rw26.c:858:ll_write_end()) pos 716800, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445969:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f9abaab1[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.445970:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000000526d9e 17ffffc0001001 3:0 ffff9d8190f4f9a0 175 lru 00000080:00200000:2.0:1775764923.445974:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000099658cb4 175: 1< 0 - - > 2< 716800 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.445975:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f9abaab1 00000080:00200000:2.0:1775764923.445976:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 176. 00000080:00200000:2.0:1775764923.445978:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 176 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.445994:0:34105:0:(rw26.c:858:ll_write_end()) pos 720896, len 4096, copied 4096 00000080:00200000:2.0:1775764923.445998:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000737c2749[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446001:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000070db5ff1 17ffffc0001001 3:0 ffff9d819000fc40 176 lru 00000080:00200000:2.0:1775764923.446006:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005b122f60 176: 1< 0 - - > 2< 720896 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446008:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000737c2749 00000080:00200000:2.0:1775764923.446009:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 177. 00000080:00200000:2.0:1775764923.446011:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 177 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446025:0:34105:0:(rw26.c:858:ll_write_end()) pos 724992, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446028:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001c79cc2e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446030:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f855706f 17ffffc0001001 3:0 ffff9d819000fa80 177 lru 00000080:00200000:2.0:1775764923.446033:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ab143c9e 177: 1< 0 - - > 2< 724992 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446034:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001c79cc2e 00000080:00200000:2.0:1775764923.446035:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 178. 00000080:00200000:2.0:1775764923.446036:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 178 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446043:0:34105:0:(rw26.c:858:ll_write_end()) pos 729088, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446045:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000012341f53[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446047:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000058d367ec 17ffffc0001001 3:0 ffff9d819000f380 178 lru 00000080:00200000:2.0:1775764923.446050:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000393f904b 178: 1< 0 - - > 2< 729088 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446052:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000012341f53 00000080:00200000:2.0:1775764923.446052:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 179. 00000080:00200000:2.0:1775764923.446053:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 179 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446060:0:34105:0:(rw26.c:858:ll_write_end()) pos 733184, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446062:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000017e451[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446064:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000e010af6 17ffffc0001001 3:0 ffff9d819000fe00 179 lru 00000080:00200000:2.0:1775764923.446066:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000098228551 179: 1< 0 - - > 2< 733184 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446068:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000017e451 00000080:00200000:2.0:1775764923.446069:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 180. 00000080:00200000:2.0:1775764923.446070:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 180 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446076:0:34105:0:(rw26.c:858:ll_write_end()) pos 737280, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446078:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c5ea0415[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446080:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b86cf47c 17ffffc0001001 3:0 ffff9d819000f1c0 180 lru 00000080:00200000:2.0:1775764923.446083:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000008a617e3 180: 1< 0 - - > 2< 737280 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446085:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c5ea0415 00000080:00200000:2.0:1775764923.446085:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 181. 00000080:00200000:2.0:1775764923.446086:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 181 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446092:0:34105:0:(rw26.c:858:ll_write_end()) pos 741376, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446094:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b5f72d3e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446096:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000eb1c0e94 17ffffc0001001 3:0 ffff9d819000f700 181 lru 00000080:00200000:2.0:1775764923.446099:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006343e130 181: 1< 0 - - > 2< 741376 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446100:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b5f72d3e 00000080:00200000:2.0:1775764923.446100:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 182. 00000080:00200000:2.0:1775764923.446101:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 182 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446108:0:34105:0:(rw26.c:858:ll_write_end()) pos 745472, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446110:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000694a040c[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446111:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ebc2894c 17ffffc0001001 3:0 ffff9d819000f620 182 lru 00000080:00200000:2.0:1775764923.446114:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ad48ce00 182: 1< 0 - - > 2< 745472 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446115:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000694a040c 00000080:00200000:2.0:1775764923.446115:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 183. 00000080:00200000:2.0:1775764923.446116:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 183 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446125:0:34105:0:(rw26.c:858:ll_write_end()) pos 749568, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446127:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000037197eed[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446129:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000de0cf2c3 17ffffc0001021 2:0 ffff9d819000f460 183 lru 00000080:00200000:2.0:1775764923.446132:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a616c5a6 183: 1< 0 - - > 2< 749568 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446133:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000037197eed 00000080:00200000:2.0:1775764923.446133:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 184. 00000080:00200000:2.0:1775764923.446134:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 184 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446143:0:34105:0:(rw26.c:858:ll_write_end()) pos 753664, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446146:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000acec529f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446149:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000015cab8ef 17ffffc0001001 3:0 ffff9d819000f0e0 184 lru 00000080:00200000:2.0:1775764923.446154:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005d74f9f2 184: 1< 0 - - > 2< 753664 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446156:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000acec529f 00000080:00200000:2.0:1775764923.446156:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 185. 00000080:00200000:2.0:1775764923.446158:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 185 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446181:0:34105:0:(rw26.c:858:ll_write_end()) pos 757760, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446186:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000099164cc3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446188:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002b050531 17ffffc0001001 3:0 ffff9d819000fb60 185 lru 00000080:00200000:2.0:1775764923.446194:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d434a675 185: 1< 0 - - > 2< 757760 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446195:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000099164cc3 00000080:00200000:2.0:1775764923.446197:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 186. 00000080:00200000:2.0:1775764923.446199:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 186 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446213:0:34105:0:(rw26.c:858:ll_write_end()) pos 761856, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446216:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000425e421f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446218:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000040764302 17ffffc0001001 3:0 ffff9d819000f9a0 186 lru 00000080:00200000:2.0:1775764923.446221:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009aa365f1 186: 1< 0 - - > 2< 761856 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446222:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000425e421f 00000080:00200000:2.0:1775764923.446223:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 187. 00000080:00200000:2.0:1775764923.446224:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 187 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446231:0:34105:0:(rw26.c:858:ll_write_end()) pos 765952, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446233:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b7e53c60[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446235:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000fe3902d 17ffffc0001001 3:0 ffff9d819000f540 187 lru 00000080:00200000:2.0:1775764923.446238:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ae856bc0 187: 1< 0 - - > 2< 765952 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446239:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b7e53c60 00000080:00200000:2.0:1775764923.446240:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 188. 00000080:00200000:2.0:1775764923.446241:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 188 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446247:0:34105:0:(rw26.c:858:ll_write_end()) pos 770048, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446249:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a0820abe[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446251:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000092ce38b8 17ffffc0001001 3:0 ffff9d819000f000 188 lru 00000080:00200000:2.0:1775764923.446254:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000015ea0d8d 188: 1< 0 - - > 2< 770048 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446255:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a0820abe 00000080:00200000:2.0:1775764923.446256:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 189. 00000080:00200000:2.0:1775764923.446257:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 189 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446264:0:34105:0:(rw26.c:858:ll_write_end()) pos 774144, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446266:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004e6d7012[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446267:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c512d440 17ffffc0001001 3:0 ffff9d819000f2a0 189 lru 00000080:00200000:2.0:1775764923.446270:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001ca8b745 189: 1< 0 - - > 2< 774144 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446271:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004e6d7012 00000080:00200000:2.0:1775764923.446271:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 190. 00000080:00200000:2.0:1775764923.446272:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 190 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446279:0:34105:0:(rw26.c:858:ll_write_end()) pos 778240, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446281:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d6a7a62c[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446282:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000be8884af 17ffffc0001001 3:0 ffff9d819000fee0 190 lru 00000080:00200000:2.0:1775764923.446285:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000032e9f9b 190: 1< 0 - - > 2< 778240 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446286:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d6a7a62c 00000080:00200000:2.0:1775764923.446287:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 191. 00000080:00200000:2.0:1775764923.446288:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 191 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446294:0:34105:0:(rw26.c:858:ll_write_end()) pos 782336, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446296:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001c59d677[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446298:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004857b176 17ffffc0001001 3:0 ffff9d819000fd20 191 lru 00000080:00200000:2.0:1775764923.446301:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002b8702fd 191: 1< 0 - - > 2< 782336 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446302:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001c59d677 00000080:00200000:2.0:1775764923.446303:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 192. 00000080:00200000:2.0:1775764923.446304:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 192 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446313:0:34105:0:(rw26.c:858:ll_write_end()) pos 786432, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446315:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000369e34ed[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446316:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007b6571bd 17ffffc0001001 3:0 ffff9d819000f7e0 192 lru 00000080:00200000:2.0:1775764923.446320:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ce893850 192: 1< 0 - - > 2< 786432 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446321:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000369e34ed 00000080:00200000:2.0:1775764923.446321:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 193. 00000080:00200000:2.0:1775764923.446322:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 193 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446329:0:34105:0:(rw26.c:858:ll_write_end()) pos 790528, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446331:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009435546c[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446333:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a9905907 17ffffc0001001 3:0 ffff9d819000f8c0 193 lru 00000080:00200000:2.0:1775764923.446336:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a7c698cb 193: 1< 0 - - > 2< 790528 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446338:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009435546c 00000080:00200000:2.0:1775764923.446338:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 194. 00000080:00200000:2.0:1775764923.446339:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 194 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446350:0:34105:0:(rw26.c:858:ll_write_end()) pos 794624, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446352:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004161058e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446353:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c5506eb5 17ffffc0001001 3:0 ffff9d8186cceb60 194 lru 00000080:00200000:2.0:1775764923.446356:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000603e9968 194: 1< 0 - - > 2< 794624 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446357:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004161058e 00000080:00200000:2.0:1775764923.446358:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 195. 00000080:00200000:2.0:1775764923.446359:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 195 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446366:0:34105:0:(rw26.c:858:ll_write_end()) pos 798720, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446368:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004dd6e9f7[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446369:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000010d2aefc 17ffffc0001001 3:0 ffff9d8186cce9a0 195 lru 00000080:00200000:2.0:1775764923.446372:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008cc2cfe1 195: 1< 0 - - > 2< 798720 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446374:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004dd6e9f7 00000080:00200000:2.0:1775764923.446374:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 196. 00000080:00200000:2.0:1775764923.446375:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 196 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446382:0:34105:0:(rw26.c:858:ll_write_end()) pos 802816, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446384:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006326d91e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446385:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ff7a7675 17ffffc0001001 3:0 ffff9d8186cce540 196 lru 00000080:00200000:2.0:1775764923.446388:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002d6a3d73 196: 1< 0 - - > 2< 802816 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446389:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006326d91e 00000080:00200000:2.0:1775764923.446390:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 197. 00000080:00200000:2.0:1775764923.446391:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 197 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446397:0:34105:0:(rw26.c:858:ll_write_end()) pos 806912, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446399:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000086005ac7[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446401:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000dcf5bdc3 17ffffc0001001 3:0 ffff9d8186cce000 197 lru 00000080:00200000:2.0:1775764923.446404:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000f2bd049 197: 1< 0 - - > 2< 806912 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446405:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000086005ac7 00000080:00200000:2.0:1775764923.446405:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 198. 00000080:00200000:2.0:1775764923.446406:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 198 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446416:0:34105:0:(rw26.c:858:ll_write_end()) pos 811008, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446418:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ef70eb49[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446419:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007796bfbf 17ffffc0001021 2:0 ffff9d8186cce2a0 198 lru 00000080:00200000:2.0:1775764923.446422:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008f8474e4 198: 1< 0 - - > 2< 811008 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446423:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ef70eb49 00000080:00200000:2.0:1775764923.446424:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 199. 00000080:00200000:2.0:1775764923.446425:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 199 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446432:0:34105:0:(rw26.c:858:ll_write_end()) pos 815104, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446435:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e492877c[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446437:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009f1b3d08 17ffffc0001001 3:0 ffff9d8186cceee0 199 lru 00000080:00200000:2.0:1775764923.446442:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007b8cafb4 199: 1< 0 - - > 2< 815104 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446444:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e492877c 00000080:00200000:2.0:1775764923.446445:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 200. 00000080:00200000:2.0:1775764923.446446:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 200 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446458:0:34105:0:(rw26.c:858:ll_write_end()) pos 819200, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446461:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e3e97f04[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446464:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000077905bb7 17ffffc0001001 3:0 ffff9d8186cced20 200 lru 00000080:00200000:2.0:1775764923.446469:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000074f4be5e 200: 1< 0 - - > 2< 819200 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446471:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e3e97f04 00000080:00200000:2.0:1775764923.446472:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 201. 00000080:00200000:2.0:1775764923.446474:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 201 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446486:0:34105:0:(rw26.c:858:ll_write_end()) pos 823296, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446490:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000016c91477[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446493:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000089906129 17ffffc0001001 3:0 ffff9d8186cce7e0 201 lru 00000080:00200000:2.0:1775764923.446498:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007f2c8fe4 201: 1< 0 - - > 2< 823296 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446500:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000016c91477 00000080:00200000:2.0:1775764923.446500:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 202. 00000080:00200000:2.0:1775764923.446502:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 202 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446514:0:34105:0:(rw26.c:858:ll_write_end()) pos 827392, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446518:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c41caf80[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446520:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003735ce4f 17ffffc0001001 3:0 ffff9d8186cce8c0 202 lru 00000080:00200000:2.0:1775764923.446525:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002c3e165d 202: 1< 0 - - > 2< 827392 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446527:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c41caf80 00000080:00200000:2.0:1775764923.446528:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 203. 00000080:00200000:2.0:1775764923.446530:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 203 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446540:0:34105:0:(rw26.c:858:ll_write_end()) pos 831488, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446542:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000addfef47[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446544:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000616b3366 17ffffc0001001 3:0 ffff9d8186ccec40 203 lru 00000080:00200000:2.0:1775764923.446547:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006eed279c 203: 1< 0 - - > 2< 831488 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446548:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000addfef47 00000080:00200000:2.0:1775764923.446549:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 204. 00000080:00200000:2.0:1775764923.446550:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 204 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446556:0:34105:0:(rw26.c:858:ll_write_end()) pos 835584, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446558:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000021c9c0[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446559:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000015b189b8 17ffffc0001001 3:0 ffff9d8186ccea80 204 lru 00000080:00200000:2.0:1775764923.446563:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a08266d1 204: 1< 0 - - > 2< 835584 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446564:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000021c9c0 00000080:00200000:2.0:1775764923.446565:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 205. 00000080:00200000:2.0:1775764923.446566:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 205 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446573:0:34105:0:(rw26.c:858:ll_write_end()) pos 839680, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446575:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000099d148cc[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446577:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000036afbcf0 17ffffc0001001 3:0 ffff9d8186cce380 205 lru 00000080:00200000:2.0:1775764923.446580:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004d89ab4c 205: 1< 0 - - > 2< 839680 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446581:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000099d148cc 00000080:00200000:2.0:1775764923.446582:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 206. 00000080:00200000:2.0:1775764923.446583:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 206 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446589:0:34105:0:(rw26.c:858:ll_write_end()) pos 843776, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446591:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004b274f27[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446592:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ae63ef90 17ffffc0001001 3:0 ffff9d8186ccee00 206 lru 00000080:00200000:2.0:1775764923.446595:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007682db63 206: 1< 0 - - > 2< 843776 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446597:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004b274f27 00000080:00200000:2.0:1775764923.446598:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 207. 00000080:00200000:2.0:1775764923.446599:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 207 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446605:0:34105:0:(rw26.c:858:ll_write_end()) pos 847872, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446607:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000083d0a713[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446609:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000130cae20 17ffffc0001001 3:0 ffff9d8186cce1c0 207 lru 00000080:00200000:2.0:1775764923.446612:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000013856a73 207: 1< 0 - - > 2< 847872 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446613:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000083d0a713 00000080:00200000:2.0:1775764923.446613:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 208. 00000080:00200000:2.0:1775764923.446614:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 208 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446621:0:34105:0:(rw26.c:858:ll_write_end()) pos 851968, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446623:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000119cd14d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446624:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ebe90a55 17ffffc0001001 3:0 ffff9d8186cce700 208 lru 00000080:00200000:2.0:1775764923.446627:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006f560246 208: 1< 0 - - > 2< 851968 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446628:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000119cd14d 00000080:00200000:2.0:1775764923.446629:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 209. 00000080:00200000:2.0:1775764923.446629:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 209 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446635:0:34105:0:(rw26.c:858:ll_write_end()) pos 856064, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446637:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000d91e06e[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446639:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000645a6078 17ffffc0001001 3:0 ffff9d8186cce620 209 lru 00000080:00200000:2.0:1775764923.446642:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000014263106 209: 1< 0 - - > 2< 856064 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446643:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000d91e06e 00000080:00200000:2.0:1775764923.446644:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 210. 00000080:00200000:2.0:1775764923.446644:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 210 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446651:0:34105:0:(rw26.c:858:ll_write_end()) pos 860160, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446652:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002f495483[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446654:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e94028ee 17ffffc0001001 3:0 ffff9d8186cce460 210 lru 00000080:00200000:2.0:1775764923.446657:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000006377a04 210: 1< 0 - - > 2< 860160 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446658:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002f495483 00000080:00200000:2.0:1775764923.446658:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 211. 00000080:00200000:2.0:1775764923.446659:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 211 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446666:0:34105:0:(rw26.c:858:ll_write_end()) pos 864256, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446668:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003a45ef9a[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446669:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a62dea8a 17ffffc0001001 3:0 ffff9d8186cce0e0 211 lru 00000080:00200000:2.0:1775764923.446672:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003c62442e 211: 1< 0 - - > 2< 864256 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446673:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003a45ef9a 00000080:00200000:2.0:1775764923.446674:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 212. 00000080:00200000:2.0:1775764923.446674:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 212 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446686:0:34105:0:(rw26.c:858:ll_write_end()) pos 868352, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446688:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c9b1b418[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446690:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000024e6d4b0 17ffffc0001001 3:0 ffff9d8190722620 212 lru 00000080:00200000:2.0:1775764923.446693:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e52e8830 212: 1< 0 - - > 2< 868352 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446694:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c9b1b418 00000080:00200000:2.0:1775764923.446695:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 213. 00000080:00200000:2.0:1775764923.446696:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 213 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446705:0:34105:0:(rw26.c:858:ll_write_end()) pos 872448, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446707:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000089ef0496[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446708:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001a040c82 17ffffc0001021 2:0 ffff9d8190722460 213 lru 00000080:00200000:2.0:1775764923.446711:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e1d21ac6 213: 1< 0 - - > 2< 872448 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446712:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000089ef0496 00000080:00200000:2.0:1775764923.446713:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 214. 00000080:00200000:2.0:1775764923.446714:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 214 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446721:0:34105:0:(rw26.c:858:ll_write_end()) pos 876544, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446723:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002e1ccab7[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446725:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c833c9cf 17ffffc0001001 3:0 ffff9d81907220e0 214 lru 00000080:00200000:2.0:1775764923.446728:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bb1412d9 214: 1< 0 - - > 2< 876544 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446729:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002e1ccab7 00000080:00200000:2.0:1775764923.446730:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 215. 00000080:00200000:2.0:1775764923.446731:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 215 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446738:0:34105:0:(rw26.c:858:ll_write_end()) pos 880640, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446740:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000371ea01[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446741:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d569b5c7 17ffffc0001001 3:0 ffff9d8190722b60 215 lru 00000080:00200000:2.0:1775764923.446744:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000147ae5c7 215: 1< 0 - - > 2< 880640 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446745:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000371ea01 00000080:00200000:2.0:1775764923.446746:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 216. 00000080:00200000:2.0:1775764923.446747:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 216 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446753:0:34105:0:(rw26.c:858:ll_write_end()) pos 884736, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446755:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fe18e1b1[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446757:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000088b34962 17ffffc0001001 3:0 ffff9d81907229a0 216 lru 00000080:00200000:2.0:1775764923.446759:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000076083bfd 216: 1< 0 - - > 2< 884736 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446760:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fe18e1b1 00000080:00200000:2.0:1775764923.446761:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 217. 00000080:00200000:2.0:1775764923.446762:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 217 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446769:0:34105:0:(rw26.c:858:ll_write_end()) pos 888832, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446771:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a634b833[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446772:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e0ffec24 17ffffc0001001 3:0 ffff9d8190722540 217 lru 00000080:00200000:2.0:1775764923.446775:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ab4c4f42 217: 1< 0 - - > 2< 888832 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446776:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a634b833 00000080:00200000:2.0:1775764923.446777:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 218. 00000080:00200000:2.0:1775764923.446777:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 218 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446784:0:34105:0:(rw26.c:858:ll_write_end()) pos 892928, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446786:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000942b4430[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446787:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a2ab9ee2 17ffffc0001001 3:0 ffff9d8190722000 218 lru 00000080:00200000:2.0:1775764923.446790:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000096de5a51 218: 1< 0 - - > 2< 892928 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446791:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000942b4430 00000080:00200000:2.0:1775764923.446792:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 219. 00000080:00200000:2.0:1775764923.446792:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 219 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446799:0:34105:0:(rw26.c:858:ll_write_end()) pos 897024, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446801:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000dc658e35[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446803:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b9abff84 17ffffc0001001 3:0 ffff9d81907222a0 219 lru 00000080:00200000:2.0:1775764923.446806:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004a9781e1 219: 1< 0 - - > 2< 897024 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446807:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000dc658e35 00000080:00200000:2.0:1775764923.446807:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 220. 00000080:00200000:2.0:1775764923.446808:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 220 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446815:0:34105:0:(rw26.c:858:ll_write_end()) pos 901120, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446817:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000069b5bd49[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446819:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f0251fe8 17ffffc0001001 3:0 ffff9d8190722ee0 220 lru 00000080:00200000:2.0:1775764923.446822:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001c6f6211 220: 1< 0 - - > 2< 901120 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446823:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000069b5bd49 00000080:00200000:2.0:1775764923.446824:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 221. 00000080:00200000:2.0:1775764923.446824:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 221 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446831:0:34105:0:(rw26.c:858:ll_write_end()) pos 905216, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446832:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000bd16673b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446834:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002c3423df 17ffffc0001001 3:0 ffff9d8190722d20 221 lru 00000080:00200000:2.0:1775764923.446837:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a37f2aca 221: 1< 0 - - > 2< 905216 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446838:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000bd16673b 00000080:00200000:2.0:1775764923.446839:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 222. 00000080:00200000:2.0:1775764923.446839:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 222 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446846:0:34105:0:(rw26.c:858:ll_write_end()) pos 909312, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446848:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000081ee23a8[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446850:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a6582b4d 17ffffc0001001 3:0 ffff9d81907227e0 222 lru 00000080:00200000:2.0:1775764923.446853:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006189626e 222: 1< 0 - - > 2< 909312 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446854:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000081ee23a8 00000080:00200000:2.0:1775764923.446854:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 223. 00000080:00200000:2.0:1775764923.446855:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 223 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446861:0:34105:0:(rw26.c:858:ll_write_end()) pos 913408, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446863:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f0ef950d[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446865:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cd80bab8 17ffffc0001001 3:0 ffff9d81907228c0 223 lru 00000080:00200000:2.0:1775764923.446868:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d9b8ac69 223: 1< 0 - - > 2< 913408 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446869:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f0ef950d 00000080:00200000:2.0:1775764923.446869:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 224. 00000080:00200000:2.0:1775764923.446870:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 224 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446889:0:34105:0:(rw26.c:858:ll_write_end()) pos 917504, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446892:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000001f09a37[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446894:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ad6e5372 17ffffc0001001 3:0 ffff9d8190722c40 224 lru 00000080:00200000:2.0:1775764923.446897:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009c1f2c0c 224: 1< 0 - - > 2< 917504 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446898:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000001f09a37 00000080:00200000:2.0:1775764923.446899:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 225. 00000080:00200000:2.0:1775764923.446900:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 225 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446908:0:34105:0:(rw26.c:858:ll_write_end()) pos 921600, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446910:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e062204b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446912:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008046b5ae 17ffffc0001001 3:0 ffff9d8190722a80 225 lru 00000080:00200000:2.0:1775764923.446915:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002f05d9a0 225: 1< 0 - - > 2< 921600 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446916:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e062204b 00000080:00200000:2.0:1775764923.446917:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 226. 00000080:00200000:2.0:1775764923.446919:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 226 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446930:0:34105:0:(rw26.c:858:ll_write_end()) pos 925696, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446934:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e85fb099[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446936:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000087fd7ce0 17ffffc0001001 3:0 ffff9d8190722380 226 lru 00000080:00200000:2.0:1775764923.446941:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000030c4cbe9 226: 1< 0 - - > 2< 925696 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446943:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e85fb099 00000080:00200000:2.0:1775764923.446944:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 227. 00000080:00200000:2.0:1775764923.446946:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 227 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446958:0:34105:0:(rw26.c:858:ll_write_end()) pos 929792, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446961:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a4b6b248[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446964:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b7cfcf94 17ffffc0001001 3:0 ffff9d8190722e00 227 lru 00000080:00200000:2.0:1775764923.446969:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a8180769 227: 1< 0 - - > 2< 929792 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.446971:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a4b6b248 00000080:00200000:2.0:1775764923.446972:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 228. 00000080:00200000:2.0:1775764923.446974:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 228 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.446990:0:34105:0:(rw26.c:858:ll_write_end()) pos 933888, len 4096, copied 4096 00000080:00200000:2.0:1775764923.446994:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002fca845b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.446997:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000016cb4fba 17ffffc0001021 2:0 ffff9d81907221c0 228 lru 00000080:00200000:2.0:1775764923.447002:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d33cd085 228: 1< 0 - - > 2< 933888 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447004:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002fca845b 00000080:00200000:2.0:1775764923.447005:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 229. 00000080:00200000:2.0:1775764923.447007:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 229 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447020:0:34105:0:(rw26.c:858:ll_write_end()) pos 937984, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447023:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000cd4c708[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447026:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003dd91dfc 17ffffc0001001 3:0 ffff9d8190722700 229 lru 00000080:00200000:2.0:1775764923.447029:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000050349a1e 229: 1< 0 - - > 2< 937984 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447030:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000cd4c708 00000080:00200000:2.0:1775764923.447031:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 230. 00000080:00200000:2.0:1775764923.447032:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 230 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447046:0:34105:0:(rw26.c:858:ll_write_end()) pos 942080, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447048:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000b0a73b2[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447049:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009e6c1a6a 17ffffc0001001 3:0 ffff9d81906d5c40 230 lru 00000080:00200000:2.0:1775764923.447053:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ab287e7e 230: 1< 0 - - > 2< 942080 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447054:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000b0a73b2 00000080:00200000:2.0:1775764923.447055:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 231. 00000080:00200000:2.0:1775764923.447056:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 231 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447062:0:34105:0:(rw26.c:858:ll_write_end()) pos 946176, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447064:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000097c30428[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447066:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d5ab166c 17ffffc0001001 3:0 ffff9d81906d5a80 231 lru 00000080:00200000:2.0:1775764923.447069:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000000e147f7 231: 1< 0 - - > 2< 946176 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447070:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000097c30428 00000080:00200000:2.0:1775764923.447070:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 232. 00000080:00200000:2.0:1775764923.447071:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 232 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447078:0:34105:0:(rw26.c:858:ll_write_end()) pos 950272, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447080:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004eb96527[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447081:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f4dbf881 17ffffc0001001 3:0 ffff9d81906d5380 232 lru 00000080:00200000:2.0:1775764923.447084:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000096afc8d7 232: 1< 0 - - > 2< 950272 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447086:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004eb96527 00000080:00200000:2.0:1775764923.447086:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 233. 00000080:00200000:2.0:1775764923.447087:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 233 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447094:0:34105:0:(rw26.c:858:ll_write_end()) pos 954368, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447096:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000002fe3bf2[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447098:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e398da07 17ffffc0001001 3:0 ffff9d81906d5e00 233 lru 00000080:00200000:2.0:1775764923.447101:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000006a4a123 233: 1< 0 - - > 2< 954368 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447102:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000002fe3bf2 00000080:00200000:2.0:1775764923.447103:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 234. 00000080:00200000:2.0:1775764923.447104:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 234 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447110:0:34105:0:(rw26.c:858:ll_write_end()) pos 958464, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447112:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008143c34b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447114:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000009314da1 17ffffc0001001 3:0 ffff9d81906d51c0 234 lru 00000080:00200000:2.0:1775764923.447117:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000926b59f7 234: 1< 0 - - > 2< 958464 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447118:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008143c34b 00000080:00200000:2.0:1775764923.447118:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 235. 00000080:00200000:2.0:1775764923.447119:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 235 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447125:0:34105:0:(rw26.c:858:ll_write_end()) pos 962560, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447127:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000fe326cf3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447129:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b0d865a3 17ffffc0001001 3:0 ffff9d81906d5700 235 lru 00000080:00200000:2.0:1775764923.447132:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003421c387 235: 1< 0 - - > 2< 962560 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447133:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000fe326cf3 00000080:00200000:2.0:1775764923.447133:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 236. 00000080:00200000:2.0:1775764923.447134:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 236 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447140:0:34105:0:(rw26.c:858:ll_write_end()) pos 966656, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447142:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004ee917aa[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447144:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f316b9af 17ffffc0001001 3:0 ffff9d81906d5620 236 lru 00000080:00200000:2.0:1775764923.447147:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006919cb31 236: 1< 0 - - > 2< 966656 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447149:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004ee917aa 00000080:00200000:2.0:1775764923.447149:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 237. 00000080:00200000:2.0:1775764923.447150:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 237 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447156:0:34105:0:(rw26.c:858:ll_write_end()) pos 970752, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447158:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c51608f3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447160:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d326b0e5 17ffffc0001001 3:0 ffff9d81906d5460 237 lru 00000080:00200000:2.0:1775764923.447163:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000add8ee2b 237: 1< 0 - - > 2< 970752 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447164:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c51608f3 00000080:00200000:2.0:1775764923.447165:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 238. 00000080:00200000:2.0:1775764923.447165:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 238 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447187:0:34105:0:(rw26.c:858:ll_write_end()) pos 974848, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447190:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004f7f606f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447191:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c529b58a 17ffffc0001001 3:0 ffff9d81906d50e0 238 lru 00000080:00200000:2.0:1775764923.447194:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001d1cf46c 238: 1< 0 - - > 2< 974848 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447196:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004f7f606f 00000080:00200000:2.0:1775764923.447196:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 239. 00000080:00200000:2.0:1775764923.447197:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 239 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447207:0:34105:0:(rw26.c:858:ll_write_end()) pos 978944, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447210:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cfa15837[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447213:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002d51c70e 17ffffc0001001 3:0 ffff9d81906d5b60 239 lru 00000080:00200000:2.0:1775764923.447218:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000064b98dbf 239: 1< 0 - - > 2< 978944 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447220:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cfa15837 00000080:00200000:2.0:1775764923.447221:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 240. 00000080:00200000:2.0:1775764923.447222:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 240 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447236:0:34105:0:(rw26.c:858:ll_write_end()) pos 983040, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447240:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000048d067c5[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447242:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002a2ecadf 17ffffc0001001 3:0 ffff9d81906d59a0 240 lru 00000080:00200000:2.0:1775764923.447247:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d7d42d72 240: 1< 0 - - > 2< 983040 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447249:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000048d067c5 00000080:00200000:2.0:1775764923.447250:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 241. 00000080:00200000:2.0:1775764923.447252:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 241 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447265:0:34105:0:(rw26.c:858:ll_write_end()) pos 987136, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447269:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000658cc5cc[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447271:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000adb928de 17ffffc0001001 3:0 ffff9d81906d5540 241 lru 00000080:00200000:2.0:1775764923.447276:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000046d2b099 241: 1< 0 - - > 2< 987136 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447278:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000658cc5cc 00000080:00200000:2.0:1775764923.447280:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 242. 00000080:00200000:2.0:1775764923.447282:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 242 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447293:0:34105:0:(rw26.c:858:ll_write_end()) pos 991232, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447295:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f35e5059[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447297:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000077454022 17ffffc0001001 3:0 ffff9d81906d5000 242 lru 00000080:00200000:2.0:1775764923.447300:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a30ed271 242: 1< 0 - - > 2< 991232 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447302:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f35e5059 00000080:00200000:2.0:1775764923.447302:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 243. 00000080:00200000:2.0:1775764923.447304:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 243 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447313:0:34105:0:(rw26.c:858:ll_write_end()) pos 995328, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447315:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007fd0daea[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447316:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f9996813 17ffffc0001021 2:0 ffff9d81906d52a0 243 lru 00000080:00200000:2.0:1775764923.447319:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000019b8c3d0 243: 1< 0 - - > 2< 995328 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447321:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007fd0daea 00000080:00200000:2.0:1775764923.447321:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 244. 00000080:00200000:2.0:1775764923.447322:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 244 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447328:0:34105:0:(rw26.c:858:ll_write_end()) pos 999424, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447330:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000032503363[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447332:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000d59303c 17ffffc0001001 3:0 ffff9d81906d5ee0 244 lru 00000080:00200000:2.0:1775764923.447335:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b95f78bd 244: 1< 0 - - > 2< 999424 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447336:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000032503363 00000080:00200000:2.0:1775764923.447336:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 245. 00000080:00200000:2.0:1775764923.447337:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 245 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447344:0:34105:0:(rw26.c:858:ll_write_end()) pos 1003520, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447346:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d27c59c4[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447348:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000619b584c 17ffffc0001001 3:0 ffff9d81906d5d20 245 lru 00000080:00200000:2.0:1775764923.447351:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d32ac24a 245: 1< 0 - - > 2< 1003520 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447352:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d27c59c4 00000080:00200000:2.0:1775764923.447353:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 246. 00000080:00200000:2.0:1775764923.447354:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 246 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447360:0:34105:0:(rw26.c:858:ll_write_end()) pos 1007616, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447363:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000041d51df6[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447364:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e3f5cfa0 17ffffc0001001 3:0 ffff9d81906d57e0 246 lru 00000080:00200000:2.0:1775764923.447367:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004418e98f 246: 1< 0 - - > 2< 1007616 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447368:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000041d51df6 00000080:00200000:2.0:1775764923.447369:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 247. 00000080:00200000:2.0:1775764923.447370:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 247 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447377:0:34105:0:(rw26.c:858:ll_write_end()) pos 1011712, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447379:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e816c089[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447381:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005f8882e1 17ffffc0001001 3:0 ffff9d81906d58c0 247 lru 00000080:00200000:2.0:1775764923.447383:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e8f2e9a6 247: 1< 0 - - > 2< 1011712 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447385:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e816c089 00000080:00200000:2.0:1775764923.447385:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 248. 00000080:00200000:2.0:1775764923.447386:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 248 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447397:0:34105:0:(rw26.c:858:ll_write_end()) pos 1015808, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447399:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000088f3eeac[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447400:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000066b440c3 17ffffc0001001 3:0 ffff9d8190276000 248 lru 00000080:00200000:2.0:1775764923.447403:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009b4908a2 248: 1< 0 - - > 2< 1015808 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447404:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000088f3eeac 00000080:00200000:2.0:1775764923.447405:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 249. 00000080:00200000:2.0:1775764923.447406:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 249 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447413:0:34105:0:(rw26.c:858:ll_write_end()) pos 1019904, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447415:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009ad03e6f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447417:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b2654ca4 17ffffc0001001 3:0 ffff9d81902762a0 249 lru 00000080:00200000:2.0:1775764923.447420:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000359c3025 249: 1< 0 - - > 2< 1019904 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447421:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009ad03e6f 00000080:00200000:2.0:1775764923.447422:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 250. 00000080:00200000:2.0:1775764923.447422:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 250 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447429:0:34105:0:(rw26.c:858:ll_write_end()) pos 1024000, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447431:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000090d6efd8[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447432:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f7827920 17ffffc0001001 3:0 ffff9d8190276ee0 250 lru 00000080:00200000:2.0:1775764923.447435:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000328e1a1b 250: 1< 0 - - > 2< 1024000 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447436:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000090d6efd8 00000080:00200000:2.0:1775764923.447437:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 251. 00000080:00200000:2.0:1775764923.447438:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 251 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447444:0:34105:0:(rw26.c:858:ll_write_end()) pos 1028096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447446:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000030015764[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447448:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e2629de8 17ffffc0001001 3:0 ffff9d8190276d20 251 lru 00000080:00200000:2.0:1775764923.447451:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bffcbc61 251: 1< 0 - - > 2< 1028096 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447453:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000030015764 00000080:00200000:2.0:1775764923.447453:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 252. 00000080:00200000:2.0:1775764923.447454:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 252 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447461:0:34105:0:(rw26.c:858:ll_write_end()) pos 1032192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447463:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000004a644e9[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447464:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a6935aa9 17ffffc0001001 3:0 ffff9d81902767e0 252 lru 00000080:00200000:2.0:1775764923.447467:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000044357729 252: 1< 0 - - > 2< 1032192 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447468:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000004a644e9 00000080:00200000:2.0:1775764923.447469:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 253. 00000080:00200000:2.0:1775764923.447471:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 253 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447482:0:34105:0:(rw26.c:858:ll_write_end()) pos 1036288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447485:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e236a991[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447487:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f30bbde2 17ffffc0001001 3:0 ffff9d81902768c0 253 lru 00000080:00200000:2.0:1775764923.447492:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000067aa00d 253: 1< 0 - - > 2< 1036288 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447494:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e236a991 00000080:00200000:2.0:1775764923.447496:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 254. 00000080:00200000:2.0:1775764923.447497:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 254 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447510:0:34105:0:(rw26.c:858:ll_write_end()) pos 1040384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447514:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000014c2a7eb[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447517:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000331bb9df 17ffffc0001001 3:0 ffff9d8190276c40 254 lru 00000080:00200000:2.0:1775764923.447521:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008d2e0d69 254: 1< 0 - - > 2< 1040384 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447522:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000014c2a7eb 00000080:00200000:2.0:1775764923.447523:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 255. 00000080:00200000:2.0:1775764923.447524:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 255 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447533:0:34105:0:(rw26.c:858:ll_write_end()) pos 1044480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447535:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004ae5f269[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447537:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008ef27252 17ffffc0001001 3:0 ffff9d8190276a80 255 lru 00000080:00200000:2.0:1775764923.447540:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000089250a15 255: 1< 0 - - > 2< 1044480 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447541:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004ae5f269 00000080:00200000:2.0:1775764923.447542:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 256. 00000080:00200000:2.0:1775764923.447543:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 256 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447551:0:34105:0:(rw26.c:858:ll_write_end()) pos 1048576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447553:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000033343739[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447555:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000086947137 17ffffc0001001 3:0 ffff9d8190276380 256 lru 00000080:00200000:2.0:1775764923.447558:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a02347f2 256: 1< 0 - - > 2< 1048576 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447559:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000033343739 00000080:00200000:2.0:1775764923.447560:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 257. 00000080:00200000:2.0:1775764923.447561:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 257 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447567:0:34105:0:(rw26.c:858:ll_write_end()) pos 1052672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447569:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000df7192e8[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447570:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000073647562 17ffffc0001001 3:0 ffff9d8190276e00 257 lru 00000080:00200000:2.0:1775764923.447573:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008e39e91b 257: 1< 0 - - > 2< 1052672 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447574:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000df7192e8 00000080:00200000:2.0:1775764923.447575:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 258. 00000080:00200000:2.0:1775764923.447576:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 258 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447584:0:34105:0:(rw26.c:858:ll_write_end()) pos 1056768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447586:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000050304ef3[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447588:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000698a77a4 17ffffc0001021 2:0 ffff9d81902761c0 258 lru 00000080:00200000:2.0:1775764923.447591:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009ed06d2a 258: 1< 0 - - > 2< 1056768 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447592:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000050304ef3 00000080:00200000:2.0:1775764923.447593:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 259. 00000080:00200000:2.0:1775764923.447594:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 259 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447600:0:34105:0:(rw26.c:858:ll_write_end()) pos 1060864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447602:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000043899776[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447603:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004642ef26 17ffffc0001001 3:0 ffff9d8190276700 259 lru 00000080:00200000:2.0:1775764923.447606:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000e6f0dd73 259: 1< 0 - - > 2< 1060864 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447607:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000043899776 00000080:00200000:2.0:1775764923.447608:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 260. 00000080:00200000:2.0:1775764923.447609:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 260 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447616:0:34105:0:(rw26.c:858:ll_write_end()) pos 1064960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447618:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003e7fdf3f[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447619:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e8866afa 17ffffc0001001 3:0 ffff9d8190276620 260 lru 00000080:00200000:2.0:1775764923.447622:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000084c7620b 260: 1< 0 - - > 2< 1064960 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447624:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003e7fdf3f 00000080:00200000:2.0:1775764923.447624:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 261. 00000080:00200000:2.0:1775764923.447626:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 261 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447633:0:34105:0:(rw26.c:858:ll_write_end()) pos 1069056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447635:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000017742af9[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447637:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f52baf04 17ffffc0001001 3:0 ffff9d8190276460 261 lru 00000080:00200000:2.0:1775764923.447640:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000345bfa6a 261: 1< 0 - - > 2< 1069056 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447641:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000017742af9 00000080:00200000:2.0:1775764923.447642:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 262. 00000080:00200000:2.0:1775764923.447642:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 262 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447649:0:34105:0:(rw26.c:858:ll_write_end()) pos 1073152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447651:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003eae2b7b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447652:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000022d41032 17ffffc0001001 3:0 ffff9d81902760e0 262 lru 00000080:00200000:2.0:1775764923.447655:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000be8ab647 262: 1< 0 - - > 2< 1073152 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447656:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003eae2b7b 00000080:00200000:2.0:1775764923.447657:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 263. 00000080:00200000:2.0:1775764923.447657:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 263 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447663:0:34105:0:(rw26.c:858:ll_write_end()) pos 1077248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447665:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009b633a53[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447667:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a3407894 17ffffc0001001 3:0 ffff9d8190276b60 263 lru 00000080:00200000:2.0:1775764923.447670:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000001020735 263: 1< 0 - - > 2< 1077248 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447671:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009b633a53 00000080:00200000:2.0:1775764923.447671:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 264. 00000080:00200000:2.0:1775764923.447672:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 264 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447679:0:34105:0:(rw26.c:858:ll_write_end()) pos 1081344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447681:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000933c9fea[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447684:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000006999593 17ffffc0001001 3:0 ffff9d81902769a0 264 lru 00000080:00200000:2.0:1775764923.447688:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003e27a005 264: 1< 0 - - > 2< 1081344 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447690:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000933c9fea 00000080:00200000:2.0:1775764923.447691:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 265. 00000080:00200000:2.0:1775764923.447692:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 265 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447704:0:34105:0:(rw26.c:858:ll_write_end()) pos 1085440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447708:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001e1e0f1b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447710:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009525dfa7 17ffffc0001001 3:0 ffff9d8190276540 265 lru 00000080:00200000:2.0:1775764923.447716:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a9b9be74 265: 1< 0 - - > 2< 1085440 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447718:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001e1e0f1b 00000080:00200000:2.0:1775764923.447719:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 266. 00000080:00200000:2.0:1775764923.447720:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 266 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447738:0:34105:0:(rw26.c:858:ll_write_end()) pos 1089536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447741:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000065c91595[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447743:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007072188f 17ffffc0001001 3:0 ffff9d819124c8c0 266 lru 00000080:00200000:2.0:1775764923.447746:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f7037514 266: 1< 0 - - > 2< 1089536 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447748:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000065c91595 00000080:00200000:2.0:1775764923.447748:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 267. 00000080:00200000:2.0:1775764923.447750:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 267 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447757:0:34105:0:(rw26.c:858:ll_write_end()) pos 1093632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447759:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000eda4b642[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447761:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c8f10ed7 17ffffc0001001 3:0 ffff9d819124cc40 267 lru 00000080:00200000:2.0:1775764923.447764:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000088c27961 267: 1< 0 - - > 2< 1093632 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447766:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000eda4b642 00000080:00200000:2.0:1775764923.447766:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 268. 00000080:00200000:2.0:1775764923.447767:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 268 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447774:0:34105:0:(rw26.c:858:ll_write_end()) pos 1097728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447776:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000095afd7af[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447778:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c826f633 17ffffc0001001 3:0 ffff9d819124ca80 268 lru 00000080:00200000:2.0:1775764923.447780:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000368fd28f 268: 1< 0 - - > 2< 1097728 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447782:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000095afd7af 00000080:00200000:2.0:1775764923.447782:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 269. 00000080:00200000:2.0:1775764923.447783:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 269 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447790:0:34105:0:(rw26.c:858:ll_write_end()) pos 1101824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447792:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b2b0e3b6[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447793:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000001fd3c8f 17ffffc0001001 3:0 ffff9d819124c380 269 lru 00000080:00200000:2.0:1775764923.447796:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007378e8cb 269: 1< 0 - - > 2< 1101824 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447798:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b2b0e3b6 00000080:00200000:2.0:1775764923.447798:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 270. 00000080:00200000:2.0:1775764923.447799:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 270 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447805:0:34105:0:(rw26.c:858:ll_write_end()) pos 1105920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447807:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000055e28752[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447809:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007e3eb2a9 17ffffc0001001 3:0 ffff9d819124ce00 270 lru 00000080:00200000:2.0:1775764923.447812:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009a9f2372 270: 1< 0 - - > 2< 1105920 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447813:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000055e28752 00000080:00200000:2.0:1775764923.447813:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 271. 00000080:00200000:2.0:1775764923.447814:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 271 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447820:0:34105:0:(rw26.c:858:ll_write_end()) pos 1110016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447822:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000072dd1aa5[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447824:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005779e45f 17ffffc0001001 3:0 ffff9d819124c1c0 271 lru 00000080:00200000:2.0:1775764923.447827:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d0288803 271: 1< 0 - - > 2< 1110016 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447828:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000072dd1aa5 00000080:00200000:2.0:1775764923.447829:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 272. 00000080:00200000:2.0:1775764923.447830:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 272 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447836:0:34105:0:(rw26.c:858:ll_write_end()) pos 1114112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447838:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f90e1140[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447839:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000966e7612 17ffffc0001001 3:0 ffff9d819124c700 272 lru 00000080:00200000:2.0:1775764923.447842:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a4d3d804 272: 1< 0 - - > 2< 1114112 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447843:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f90e1140 00000080:00200000:2.0:1775764923.447844:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 273. 00000080:00200000:2.0:1775764923.447845:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 273 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447853:0:34105:0:(rw26.c:858:ll_write_end()) pos 1118208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447855:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b031b37b[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447857:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000054e3604 17ffffc0001021 2:0 ffff9d819124c620 273 lru 00000080:00200000:2.0:1775764923.447860:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001acbf852 273: 1< 0 - - > 2< 1118208 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447861:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b031b37b 00000080:00200000:2.0:1775764923.447861:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 274. 00000080:00200000:2.0:1775764923.447862:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 274 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447869:0:34105:0:(rw26.c:858:ll_write_end()) pos 1122304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447871:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004bfec948[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447873:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000000b2519b7 17ffffc0001001 3:0 ffff9d819124c460 274 lru 00000080:00200000:2.0:1775764923.447889:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000babc13df 274: 1< 0 - - > 2< 1122304 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447890:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004bfec948 00000080:00200000:2.0:1775764923.447891:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 275. 00000080:00200000:2.0:1775764923.447892:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 275 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447903:0:34105:0:(rw26.c:858:ll_write_end()) pos 1126400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447905:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000727a1b18[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447907:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b1eea4b4 17ffffc0001001 3:0 ffff9d819124c0e0 275 lru 00000080:00200000:2.0:1775764923.447910:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dc2249a7 275: 1< 0 - - > 2< 1126400 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447912:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000727a1b18 00000080:00200000:2.0:1775764923.447913:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 276. 00000080:00200000:2.0:1775764923.447914:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 276 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447922:0:34105:0:(rw26.c:858:ll_write_end()) pos 1130496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447924:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007f827ab7[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447926:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009cb538fd 17ffffc0001001 3:0 ffff9d819124cb60 276 lru 00000080:00200000:2.0:1775764923.447929:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d7e25c08 276: 1< 0 - - > 2< 1130496 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447931:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007f827ab7 00000080:00200000:2.0:1775764923.447932:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 277. 00000080:00200000:2.0:1775764923.447933:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 277 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447940:0:34105:0:(rw26.c:858:ll_write_end()) pos 1134592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447942:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000e8118cf9[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447944:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002e8ce023 17ffffc0001001 3:0 ffff9d819124c9a0 277 lru 00000080:00200000:2.0:1775764923.447947:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a9246102 277: 1< 0 - - > 2< 1134592 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447948:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000e8118cf9 00000080:00200000:2.0:1775764923.447949:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 278. 00000080:00200000:2.0:1775764923.447950:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 278 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.447957:0:34105:0:(rw26.c:858:ll_write_end()) pos 1138688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.447959:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c37e5c78[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447960:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000732719ea 17ffffc0001001 3:0 ffff9d819124c540 278 lru 00000080:00200000:2.0:1775764923.447963:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d375324a 278: 1< 0 - - > 2< 1138688 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447964:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c37e5c78 00000080:00200000:2.0:1775764923.447965:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 279. 00000080:00200000:2.0:1775764923.447966:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 279 of 0 to 2465 bytes 00000080:00200000:2.0:1775764923.447977:0:34105:0:(rw26.c:858:ll_write_end()) pos 1142784, len 2465, copied 2465 00000080:00200000:2.0:1775764923.447981:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000970bb55[3 00000000cb6ca502 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.447983:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000020ee7884 17ffffc0001001 3:0 ffff9d819124c000 279 lru 00000080:00200000:2.0:1775764923.447988:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000079c9dc66 279: 1< 0 - - > 2< 1142784 0 0 0x0 0x0 | 000000009452db5f 00000000b75dc773 > 3< 0 0 > 4< 0 0 8 1396736 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.447990:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000970bb55 00000080:00200000:2.0:1775764923.447991:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 280. 00000080:00200000:2.0:1775764923.447994:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 280, from 0, to 2465 prio 0 00000008:00010000:2.0:1775764923.448017:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448034:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448044:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448052:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448060:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448068:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448082:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448090:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448098:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448103:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448107:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448112:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448116:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448121:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448125:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448134:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448138:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448143:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448147:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448151:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448155:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448159:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448163:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448167:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448187:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448192:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448197:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448201:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448205:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448209:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448213:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448219:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448222:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448229:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448233:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448237:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448242:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448245:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448250:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448253:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448257:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448262:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448266:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448271:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448275:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448279:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448283:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448289:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448291:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448295:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448299:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448306:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448311:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448315:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448319:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448323:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448327:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448331:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448335:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448339:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448344:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448348:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448352:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448358:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448361:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448365:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448369:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448373:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448377:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448384:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448389:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448393:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448397:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448401:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448405:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448409:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448413:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448417:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448421:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448428:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448430:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448434:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448438:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448442:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448446:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448450:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448454:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448461:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448465:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448469:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448476:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448483:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448490:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448498:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448505:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448516:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448522:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448531:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448539:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448546:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448551:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448555:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448559:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448563:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448567:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448574:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448579:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448583:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448587:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448591:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448595:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448601:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448604:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448608:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448615:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448620:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448624:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448628:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448632:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448636:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448640:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448644:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448648:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448652:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448657:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448661:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448665:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448670:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448673:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448677:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448681:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448685:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448692:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448696:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448700:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448704:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448709:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448712:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448717:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448720:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448725:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448730:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448734:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448740:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448742:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448746:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448750:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448754:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448758:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448762:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448769:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448773:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448777:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448781:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448785:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448789:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448793:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448797:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448801:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448807:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448811:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448818:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448826:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448834:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448842:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448850:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448857:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448865:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448895:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448900:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448904:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448908:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448912:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448916:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448920:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.448927:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.448930:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448937:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448942:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448946:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448950:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448954:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448958:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448962:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448966:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448970:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448976:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448984:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.448992:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449001:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449008:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.449020:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.449025:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449033:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449038:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449046:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449053:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449061:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449069:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449078:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449086:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449095:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449101:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449106:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449111:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449115:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449120:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.449126:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.449129:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449133:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449137:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449142:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449146:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449153:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449158:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449162:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449166:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449183:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449188:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449192:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449196:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449200:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449205:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.449211:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.449214:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449218:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449223:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449227:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449231:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449235:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449239:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449246:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449250:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449254:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449258:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449262:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449266:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449270:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449274:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.449280:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.449283:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449288:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449292:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449296:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449300:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449304:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449308:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449312:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449316:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449323:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449330:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449337:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449344:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449352:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449360:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.449372:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.449377:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449386:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449394:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449398:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449402:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449406:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449410:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449414:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449418:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449422:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449426:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449431:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449436:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449440:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449444:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.449450:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.449453:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449457:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449461:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449465:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449476:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449484:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449492:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449500:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449508:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449516:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449521:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449525:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449529:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449534:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449538:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.449545:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 15, dirtied 15 00000008:00010000:2.0:1775764923.449547:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449552:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449556:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449560:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449564:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449568:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449576:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449580:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449584:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.449588:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000a89c3596 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.449593:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fabb8, count 10, dirtied 10 00000080:00200000:2.0:1775764923.449596:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 280 pages 1145249 bytes, tot: 1145249 00000080:00200000:2.0:1775764923.449598:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sanity.sh: write bytes 0, result: 1145249 00000020:00010000:2.0:1775764923.449606:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.449607:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.449609:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cedb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.449613:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.449622:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.449631:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.449634:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cedb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.449640:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 1145249 bytes 00000080:00200000:2.0:1775764923.449644:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 279] 00000080:00200000:2.0:1775764923.449663:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x48:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.449667:0:34105:0:(file.c:2206:ll_file_io_generic()) sanity.sh: 2 io complete with rc: 0, result: 1145249, restart: 0 00000080:00200000:2.0:1775764923.449673:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x48:0x0] updating i_size 1145249 i_blocks 0 00000080:00200000:2.0:1775764923.449676:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 1145249 00000080:00200008:2.0:1775764923.449701:0:34105:0:(file.c:447:ll_file_release()) START file sanity.sh:[0x240004281:0x48:0x0](ffff9d81843faa08), flags 100001 00000002:00100000:2.0:1775764923.449726:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fed80 x1862023066216832/t38654705901(38654705901) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.449733:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.449736:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fd880 x1862023066217472/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.449745:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd880 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217472:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.449785:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81917656c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.450599:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd880 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217472:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.450611:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fed80 x1862023066216832/t38654705901(38654705901) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.450639:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.450647:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.450655:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.450662:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.450666:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.450670:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.450672:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.450675:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.450677:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.450681:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.450686:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.450689:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.450691:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.450693:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.450695:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.450696:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sanity.sh, flags=8 00000080:00200008:2.0:1775764923.450699:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sanity.sh:[0x240004281:0x48:0x0](ffff9d81843faa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.450702:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x48:0x0](ffff9d81843faa08),name=sanity.sh 00000002:00010000:2.0:1775764923.450708:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x48:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.450712:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.450716:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.450720:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.450722:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x48:0x0](ffff9d81843faa08) 00000080:00010000:2.0:1775764923.450723:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x48:0x0](ffff9d81843faa08) for lock 0xeef44110eff570a 00000080:00010000:2.0:1775764923.450725:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff570a from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.450727:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.450731:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.450748:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x48:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.450753:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x48:0x0] 00000008:00010000:2.0:1775764923.450763:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.450765:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cedb0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.450772:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.450778:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.450784:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.450786:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.450790:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e04195a8/0xeef44110eff5718 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1146879) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155aff expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.450794:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.450797:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x48:0x0] updating i_size 1145249 i_blocks 0 00000020:00010000:2.0:1775764923.450799:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.450801:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.450802:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cedb0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.450804:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.450806:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cedb0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.450813:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x48:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.450840:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.450845:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.450848:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.450853:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.450857:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.450860:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.450861:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.450863:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.450865:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.450869:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.450873:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.450894:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.450896:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.450898:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.450899:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.450937:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=scrub-performance.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.450940:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=scrub-performance.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.450946:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: scrub-performance.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.450962:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.450966:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.450972:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.450979:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217600:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.451006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81917656c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.451831:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217600:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.451841:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451848:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451852:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451856:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451860:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451863:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364e00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.451866:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451871:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451889:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451893:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.451897:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.451900:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff571f lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.451908:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fd180 x1862023066217600/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.451915:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.451925:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.451944:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.451949:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.451953:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.451960:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.451967:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.451970:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.451973:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.451977:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.451980:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.451987:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.451996:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.452001:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.452004:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.452007:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.452010:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.452017:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=scrub-performance.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.452020:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=scrub-performance.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.452027:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: scrub-performance.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.452052:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452057:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452061:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.452069:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217728:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.452096:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81917656c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.452759:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217728:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.452765:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452772:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452776:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452779:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452783:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452786:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.452787:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452790:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452794:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452797:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.452799:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452803:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5726 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.452809:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fd180 x1862023066217728/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.452818:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.452825:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.452855:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.452859:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.452863:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.452868:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.452873:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.452888:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.452890:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.452892:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.452894:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.452898:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.452903:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.452905:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.452907:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.452909:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.452910:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.452916:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.452918:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=scrub-performance.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da180, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:2.0:1775764923.452922:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=scrub-performance.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.452931:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: scrub-performance.sh,[0x240004281:0x49:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.452951:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.452955:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.452959:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.452962:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fd180 x1862023066217984/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.452967:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.452975:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fd180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217984:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.453048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81917656c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.454040:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fd180 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066217984:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.454048:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454057:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 4/0,1 mode: --/PR res: [0x240004281:0x49:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454061:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 4/0,1 mode: --/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454067:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454073:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454079:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454083:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454087:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.454092:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fd180 x1862023066217984/t38654705903(38654705903) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.454099:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fd180 x1862023066217984/t38654705903(38654705903) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.454106:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.454111:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454116:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/73 (0 0) 00000080:00200000:2.0:1775764923.454119:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.454123:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=scrub-performance.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.454152:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x49:0x0](ffff9d81843fd348) 00000080:00200000:2.0:1775764923.454227:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x49:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.454236:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x49:0x0](ffff9d81843fd348) 00000080:00010000:2.0:1775764923.454239:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x49:0x0](ffff9d81843fd348) for lock 0xeef44110eff572d 00000080:00010000:2.0:1775764923.454245:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x49:0x0](ffff9d81843fd348) 00000080:00200008:2.0:1775764923.454251:0:34105:0:(file.c:1057:ll_file_open()) START file scrub-performance.sh:[0x240004281:0x49:0x0](ffff9d81843fd348), flags 100301 00000002:00100000:2.0:1775764923.454270:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fd180 x1862023066217984/t38654705903(38654705903) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.454297:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff572d from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.454301:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454307:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.454317:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x49:0x0](ffff9d81843fd348), inode mode 81a4 mask 40 00000080:00200008:2.0:1775764923.454327:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file scrub-performance.sh:[0x240004281:0x49:0x0](ffff9d81843fd348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.454331:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x49:0x0](ffff9d81843fd348),name=scrub-performance.sh 00000002:00010000:2.0:1775764923.454339:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x49:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.454345:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.454353:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.454358:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.454362:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x49:0x0](ffff9d81843fd348) 00000080:00010000:2.0:1775764923.454365:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x49:0x0](ffff9d81843fd348) for lock 0xeef44110eff572d 00000080:00010000:2.0:1775764923.454368:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff572d from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.454371:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.454378:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.454395:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x49:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.454398:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x49:0x0] 00000008:00010000:2.0:1775764923.454407:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.454408:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce3b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.454425:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.454431:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.454450:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1775764923.455122:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.455132:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.455139:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.455145:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.455152:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.455159:0:10947:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.455177:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.455190:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.455192:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x49:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.455197:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.455202:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00010000:00010000:1.0:1775764923.455206:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1775764923.455206:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce3b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.455208:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.455213:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce3b0 (ffff9d8191255518/1) at cl_lock_fini():72 00010000:00010000:1.0:1775764923.455222:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.455229:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x49:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1775764923.455232:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fe680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066218112:192.168.202.105@tcp:101:cp.0 00000080:00200008:2.0:1775764923.455264:0:34105:0:(file.c:2612:do_file_write_iter()) START file scrub-performance.sh:[0x240004281:0x49:0x0], ppos: 0, count: 5105 00000080:00200000:2.0:1775764923.455280:0:34105:0:(file.c:2058:ll_file_io_generic()) scrub-performance.sh: write ppos: 0, bytes: 5105 00000080:00200000:2.0:1775764923.455284:0:34105:0:(file.c:1815:ll_io_set_mirror()) scrub-performance.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.455289:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@000000007b753eeb[0x110, 1, [0x240004281:0x49:0x0] hash] 00000020:00200000:2.0:1775764923.455291:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 5105) 0 0 00000080:00200000:2.0:1775764923.455293:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x49:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.455300:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 1] 00020000:00200000:2.0:1775764923.455303:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 5105) 5105, 5105 00020000:00200000:2.0:1775764923.455306:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.455317:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 1] 00000008:00010000:2.0:1775764923.455322:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.455324:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce3b0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.455329:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/549 (0 8191) 00010000:00010000:2.0:1775764923.455341:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.455346:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.455351:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.455360:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fdc00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066218240:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.455384:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4780 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.455963:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764923.456008:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.456017:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082330112 00000100:00100000:1.0:1775764923.456020:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082330112 00000100:00100000:1.0:1775764923.456046:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 299 00000100:00100000:1.0:1775764923.456051:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796d050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082330112:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.456059:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.456095:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.456101:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796d050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25526:x1862023082330112:12345-192.168.202.105@tcp:104: Request processed in 49us (139us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.456107:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 299 00010000:00010000:2.0:1775764923.456115:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.456121:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.456134:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190365600 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.456138:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.456163:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x49:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.456177:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.456207:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.456217:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.456232:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.456239:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000006b58b17a/0xeef44110eff5734 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155b53 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1775764923.456243:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066218368:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.456261:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.456263:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.456266:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.456670:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fdc00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066218240:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.456679:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.456685:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.456691:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.457022:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082330240 00000100:00100000:3.0:1775764923.457026:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082330240 00000100:00100000:3.0:1775764923.457049:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 300 00000100:00100000:3.0:1775764923.457053:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901afb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:22025:x1862023082330240:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.457059:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.457066:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->8191] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.457072:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.457077:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.457082:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.457091:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.457103:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d8190365800) 00010000:00010000:2.0:1775764923.457116:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.457123:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901afb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:22025:x1862023082330240:12345-192.168.202.105@tcp:105: Request processed in 68us (133us total) trans 0 rc 0/0 00000100:00100000:3.0:1775764923.457136:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 300 00010000:00010000:2.0:1775764923.457137:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.457146:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.457153:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.457160:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.457164:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.457185:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) scrub-performance.sh: write [0, 5105) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.457193:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.457214:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.457220:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000059eb9fa2[3 00000000ceb47a40 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.457222:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000479696e9 17ffffc0001001 3:0 ffff9d819124c2a0 0 lru 00000080:00200000:2.0:1775764923.457226:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f3eac674 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 00000000020861e5 > 3< 0 0 > 4< 0 0 8 2920448 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.457228:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000059eb9fa2 00000080:00200000:2.0:1775764923.457229:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.457233:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 1009 bytes 00000080:00200000:2.0:1775764923.457249:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 1009, copied 1009 00000080:00200000:2.0:1775764923.457253:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004639303c[3 00000000ceb47a40 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.457256:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000005dde836 17ffffc0001001 3:0 ffff9d819124cee0 1 lru 00000080:00200000:2.0:1775764923.457262:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bce41146 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 00000000020861e5 > 3< 0 0 > 4< 0 0 8 2920448 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.457264:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004639303c 00000080:00200000:2.0:1775764923.457265:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.457268:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 2, from 0, to 1009 prio 0 00000008:00010000:2.0:1775764923.457279:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 0000000045d9e4f5 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.457292:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 0000000045d9e4f5 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.457305:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fd4f8, count 2, dirtied 2 00000080:00200000:2.0:1775764923.457308:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 2 pages 5105 bytes, tot: 5105 00000080:00200000:2.0:1775764923.457310:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) scrub-performance.sh: write bytes 0, result: 5105 00000020:00010000:2.0:1775764923.457317:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.457319:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.457321:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce3b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.457323:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.457331:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.457340:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.457343:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce3b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.457348:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 5105 bytes 00000080:00200000:2.0:1775764923.457352:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 1] 00000080:00200000:2.0:1775764923.457366:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x49:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.457369:0:34105:0:(file.c:2206:ll_file_io_generic()) scrub-performance.sh: 2 io complete with rc: 0, result: 5105, restart: 0 00000080:00200000:2.0:1775764923.457375:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x49:0x0] updating i_size 5105 i_blocks 0 00000080:00200000:2.0:1775764923.457385:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 5105 00000080:00200008:2.0:1775764923.457406:0:34105:0:(file.c:447:ll_file_release()) START file scrub-performance.sh:[0x240004281:0x49:0x0](ffff9d81843fd348), flags 100001 00000002:00100000:2.0:1775764923.457429:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fd180 x1862023066217984/t38654705903(38654705903) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.457436:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.457439:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fdc00 x1862023066218624/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.457448:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fdc00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066218624:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.457478:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4780 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.457706:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066218368:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.458190:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fdc00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066218624:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.458202:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fd180 x1862023066217984/t38654705903(38654705903) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.458227:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.458234:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.458240:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.458247:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.458254:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.458257:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.458260:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.458264:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.458266:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.458273:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.458282:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.458286:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.458290:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.458293:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.458295:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.458298:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=scrub-performance.sh, flags=8 00000080:00200008:2.0:1775764923.458302:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file scrub-performance.sh:[0x240004281:0x49:0x0](ffff9d81843fd348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.458306:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x49:0x0](ffff9d81843fd348),name=scrub-performance.sh 00000002:00010000:2.0:1775764923.458314:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x49:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.458320:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.458328:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.458332:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.458335:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x49:0x0](ffff9d81843fd348) 00000080:00010000:2.0:1775764923.458337:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x49:0x0](ffff9d81843fd348) for lock 0xeef44110eff572d 00000080:00010000:2.0:1775764923.458339:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff572d from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.458341:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.458345:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.458361:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x49:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.458365:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x49:0x0] 00000008:00010000:2.0:1775764923.458375:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.458376:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce3b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.458382:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.458388:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.458393:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.458395:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.458400:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000e85d652a/0xeef44110eff573b lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155b5a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.458404:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.458407:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x49:0x0] updating i_size 5105 i_blocks 0 00000020:00010000:2.0:1775764923.458409:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.458411:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.458412:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce3b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.458414:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.458415:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce3b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.458424:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x49:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.458455:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.458462:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.458467:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.458474:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.458479:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.458482:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.458484:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.458487:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.458490:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.458496:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.458502:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.458505:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.458507:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.458509:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.458511:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.458518:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup-cifs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.458521:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-cifs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.458526:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup-cifs.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.458544:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.458550:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.458556:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.458566:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ffb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066218752:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.458595:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4780 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.459422:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ffb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066218752:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.459428:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459434:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459437:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459441:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459444:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459447:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190365a00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.459449:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459453:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459456:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459460:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.459463:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459466:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365a00/0xeef44110eff5742 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.459473:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906ffb80 x1862023066218752/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.459479:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.459486:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.459502:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.459507:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.459510:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.459515:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.459519:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.459522:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.459523:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.459525:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.459527:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.459531:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.459536:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.459539:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.459541:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.459542:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.459544:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.459548:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup-cifs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.459550:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-cifs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.459554:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup-cifs.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.459565:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459569:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.459573:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.459580:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ffb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066218880:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.459607:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4780 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.460369:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ffb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066218880:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.460378:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460385:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460392:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460397:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460403:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460408:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364200 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.460411:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460416:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460422:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460427:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.460431:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460437:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364200/0xeef44110eff5749 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.460448:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906ffb80 x1862023066218880/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.460457:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.460466:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.460497:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.460502:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.460507:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.460512:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.460516:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.460519:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.460520:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.460523:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.460524:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.460529:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.460533:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.460535:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.460538:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.460539:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.460541:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.460546:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.460548:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=setup-cifs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851dac00, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:2.0:1775764923.460552:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-cifs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.460560:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup-cifs.sh,[0x240004281:0x4a:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.460576:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.460580:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.460585:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.460589:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906ffb80 x1862023066219136/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.460594:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.460601:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906ffb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066219136:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.460657:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4780 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.461705:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906ffb80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066219136:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.461713:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461722:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461726:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461732:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461737:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461742:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461749:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461753:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.461757:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906ffb80 x1862023066219136/t38654705905(38654705905) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.461764:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906ffb80 x1862023066219136/t38654705905(38654705905) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.461771:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.461776:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461780:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/74 (0 0) 00000080:00200000:2.0:1775764923.461783:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.461787:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=setup-cifs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.461801:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x4a:0x0](ffff9d81843fdb88) 00000080:00200000:2.0:1775764923.461849:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x4a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.461855:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x4a:0x0](ffff9d81843fdb88) 00000080:00010000:2.0:1775764923.461857:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4a:0x0](ffff9d81843fdb88) for lock 0xeef44110eff5750 00000080:00010000:2.0:1775764923.461861:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4a:0x0](ffff9d81843fdb88) 00000080:00200008:2.0:1775764923.461865:0:34105:0:(file.c:1057:ll_file_open()) START file setup-cifs.sh:[0x240004281:0x4a:0x0](ffff9d81843fdb88), flags 100301 00000002:00100000:2.0:1775764923.461874:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906ffb80 x1862023066219136/t38654705905(38654705905) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.461910:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5750 from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.461913:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.461918:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.461926:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x4a:0x0](ffff9d81843fdb88), inode mode 81a4 mask 40 00000080:00200008:2.0:1775764923.461934:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file setup-cifs.sh:[0x240004281:0x4a:0x0](ffff9d81843fdb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.461938:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4a:0x0](ffff9d81843fdb88),name=setup-cifs.sh 00000002:00010000:2.0:1775764923.461946:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4a:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.461952:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.461960:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.461967:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.461970:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4a:0x0](ffff9d81843fdb88) 00000080:00010000:2.0:1775764923.461978:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4a:0x0](ffff9d81843fdb88) for lock 0xeef44110eff5750 00000080:00010000:2.0:1775764923.461982:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5750 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.461985:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.461992:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.462009:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.462013:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4a:0x0] 00000008:00010000:2.0:1775764923.462023:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.462024:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce930, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.462056:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.462062:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.462079:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.462108:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066219264:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.462794:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.462803:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.462810:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.462822:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.462829:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.462835:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.462841:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.462845:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.462850:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.462859:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.462862:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4a:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.462865:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000100:00100000:1.0:1775764923.462867:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066219264:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.462870:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.462889:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce930 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.462893:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.462894:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce930 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.462914:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.462943:0:34105:0:(file.c:2612:do_file_write_iter()) START file setup-cifs.sh:[0x240004281:0x4a:0x0], ppos: 0, count: 2475 00000080:00200000:2.0:1775764923.462949:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 2475 bytes 00000080:00200000:2.0:1775764923.462953:0:34105:0:(file.c:2588:ll_do_tiny_write()) result: 0, original count 2475 00000080:00200000:2.0:1775764923.462963:0:34105:0:(file.c:2058:ll_file_io_generic()) setup-cifs.sh: write ppos: 0, bytes: 2475 00000080:00200000:2.0:1775764923.462967:0:34105:0:(file.c:1815:ll_io_set_mirror()) setup-cifs.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.462971:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000a4b57de1[0x110, 1, [0x240004281:0x4a:0x0] hash] 00000020:00200000:2.0:1775764923.462973:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 2475) 0 0 00000080:00200000:2.0:1775764923.462975:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4a:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.462981:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1775764923.462985:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 2475) 2475, 2475 00020000:00200000:2.0:1775764923.462989:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.463004:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1775764923.463011:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.463014:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce930, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.463021:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/550 (0 4095) 00010000:00010000:2.0:1775764923.463045:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.463051:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.463056:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 3/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.463065:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fe680 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066219392:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.463087:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.463686:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764923.463720:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082330368 00000100:00100000:1.0:1775764923.463724:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082330368 00000100:00100000:1.0:1775764923.463750:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 301 00000100:00100000:1.0:1775764923.463756:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082330368:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.463763:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.463792:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.463795:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070ad80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25527:x1862023082330368:12345-192.168.202.105@tcp:104: Request processed in 40us (110us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.463800:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 301 00010000:00010000:2.0:1775764923.463815:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.463820:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.463826:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190365c00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.463829:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.463854:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.463858:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.463905:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.463914:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.463929:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.463934:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000d1f04994/0xeef44110eff5757 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155bae expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.463943:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.463945:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.463947:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.463953:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066219520:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.464412:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fe680 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066219392:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.464423:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.464431:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.464439:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.464678:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082330496 00000100:00100000:3.0:1775764923.464683:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082330496 00000100:00100000:3.0:1775764923.464709:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 302 00000100:00100000:3.0:1775764923.464716:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901af800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:39755:x1862023082330496:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.464724:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 5/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.464735:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 5/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.464744:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 5/0,1 mode: --/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.464752:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.464766:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.464774:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.464790:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d8190365200) 00010000:00010000:2.0:1775764923.464810:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.464815:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901af800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:39755:x1862023082330496:12345-192.168.202.105@tcp:105: Request processed in 99us (208us total) trans 0 rc 0/0 00000100:00100000:3.0:1775764923.464827:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 302 00010000:00010000:2.0:1775764923.464829:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.464846:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.464856:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.464866:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.464874:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.464906:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) setup-cifs.sh: write [0, 2475) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.464912:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 2475 bytes 00000080:00200000:2.0:1775764923.464936:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 2475, copied 2475 00000080:00200000:2.0:1775764923.464941:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f960bf68[3 00000000cf078c06 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.464944:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e07a8b84 17ffffc0001001 3:0 ffff9d819124cd20 0 lru 00000080:00200000:2.0:1775764923.464948:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000002152859c 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 00000000f89fca2f > 3< 0 0 > 4< 0 0 8 225280 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.464950:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f960bf68 00000080:00200000:2.0:1775764923.464951:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.464953:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 1, from 0, to 2475 prio 0 00000008:00010000:2.0:1775764923.464962:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000dc5975af ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.464971:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fdd38, count 1, dirtied 1 00000080:00200000:2.0:1775764923.464974:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 1 pages 2475 bytes, tot: 2475 00000080:00200000:2.0:1775764923.464976:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) setup-cifs.sh: write bytes 0, result: 2475 00000020:00010000:2.0:1775764923.464985:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.464988:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.464990:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce930 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.464994:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.465003:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.465011:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.465014:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce930 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.465018:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 2475 bytes 00000080:00200000:2.0:1775764923.465022:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1775764923.465035:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4a:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.465038:0:34105:0:(file.c:2206:ll_file_io_generic()) setup-cifs.sh: 2 io complete with rc: 0, result: 2475, restart: 0 00000080:00200000:2.0:1775764923.465048:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4a:0x0] updating i_size 2475 i_blocks 0 00000080:00200000:2.0:1775764923.465050:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 2475 00000080:00200008:2.0:1775764923.465066:0:34105:0:(file.c:447:ll_file_release()) START file setup-cifs.sh:[0x240004281:0x4a:0x0](ffff9d81843fdb88), flags 100001 00000002:00100000:2.0:1775764923.465086:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906ffb80 x1862023066219136/t38654705905(38654705905) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.465092:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.465095:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fe680 x1862023066219776/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.465103:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fe680 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066219776:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.465131:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.465395:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066219520:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.465818:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fe680 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066219776:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.465830:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906ffb80 x1862023066219136/t38654705905(38654705905) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.465856:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.465862:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.465869:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.465890:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.465896:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.465899:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.465901:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.465904:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.465906:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.465911:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.465917:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.465919:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.465922:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.465924:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.465926:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.465928:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=setup-cifs.sh, flags=8 00000080:00200008:2.0:1775764923.465931:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file setup-cifs.sh:[0x240004281:0x4a:0x0](ffff9d81843fdb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.465934:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4a:0x0](ffff9d81843fdb88),name=setup-cifs.sh 00000002:00010000:2.0:1775764923.465939:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4a:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.465944:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.465949:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.465953:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.465955:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4a:0x0](ffff9d81843fdb88) 00000080:00010000:2.0:1775764923.465956:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4a:0x0](ffff9d81843fdb88) for lock 0xeef44110eff5750 00000080:00010000:2.0:1775764923.465958:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5750 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.465959:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.465964:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.465978:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.465982:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4a:0x0] 00000008:00010000:2.0:1775764923.465991:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.465993:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce930, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.465999:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.466006:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.466011:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.466013:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.466018:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000b4a95abe/0xeef44110eff575e lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155bb5 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.466023:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.466025:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4a:0x0] updating i_size 2475 i_blocks 0 00000020:00010000:2.0:1775764923.466027:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.466029:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.466030:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce930 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.466032:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.466034:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce930 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.466042:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.466068:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.466074:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.466078:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.466083:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.466086:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.466089:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.466090:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.466092:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.466094:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.466102:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.466107:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.466109:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.466111:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.466112:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.466114:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.466127:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup_kerberos.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.466130:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup_kerberos.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.466135:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup_kerberos.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.466151:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466157:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466165:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.466191:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066219904:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.466228:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.466928:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066219904:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.466935:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466940:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466944:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466948:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466951:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466954:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.466957:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466960:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466963:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466967:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.466970:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.466973:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff5765 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.466980:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fce00 x1862023066219904/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.466986:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.466994:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.467010:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.467015:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.467019:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.467024:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.467028:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.467030:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.467031:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.467033:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.467035:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.467039:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.467044:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.467046:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.467048:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.467050:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.467051:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.467055:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup_kerberos.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.467057:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup_kerberos.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.467062:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup_kerberos.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.467073:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467080:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467084:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.467091:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066220032:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.467118:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.467786:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066220032:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.467792:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467797:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467801:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467804:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467807:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467810:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364e00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.467812:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467815:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467819:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467822:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.467824:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467827:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364e00/0xeef44110eff576c lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.467833:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fce00 x1862023066220032/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.467839:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.467846:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.467891:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.467897:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.467902:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.467907:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.467912:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.467915:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.467917:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.467919:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.467921:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.467929:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.467934:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.467936:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.467938:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.467940:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.467941:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.467946:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.467948:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=setup_kerberos.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da000, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.467952:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup_kerberos.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.467961:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup_kerberos.sh,[0x240004281:0x4b:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.467983:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.467990:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.467998:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.468003:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fce00 x1862023066220288/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.468011:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.468023:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066220288:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.468092:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.469079:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fce00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066220288:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.469090:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469103:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469110:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469118:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469124:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469130:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469134:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469138:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.469143:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fce00 x1862023066220288/t38654705907(38654705907) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.469150:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fce00 x1862023066220288/t38654705907(38654705907) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.469157:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.469166:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469185:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/75 (0 0) 00000080:00200000:2.0:1775764923.469189:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.469195:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=setup_kerberos.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.469213:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x4b:0x0](ffff9d81843ff448) 00000080:00200000:2.0:1775764923.469281:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x4b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.469287:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x4b:0x0](ffff9d81843ff448) 00000080:00010000:2.0:1775764923.469289:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4b:0x0](ffff9d81843ff448) for lock 0xeef44110eff5773 00000080:00010000:2.0:1775764923.469293:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4b:0x0](ffff9d81843ff448) 00000080:00200008:2.0:1775764923.469298:0:34105:0:(file.c:1057:ll_file_open()) START file setup_kerberos.sh:[0x240004281:0x4b:0x0](ffff9d81843ff448), flags 100301 00000002:00100000:2.0:1775764923.469307:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fce00 x1862023066220288/t38654705907(38654705907) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.469328:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5773 from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.469331:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469336:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.469344:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x4b:0x0](ffff9d81843ff448), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.469353:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file setup_kerberos.sh:[0x240004281:0x4b:0x0](ffff9d81843ff448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.469356:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4b:0x0](ffff9d81843ff448),name=setup_kerberos.sh 00000002:00010000:2.0:1775764923.469363:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4b:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.469367:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.469372:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.469376:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.469378:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4b:0x0](ffff9d81843ff448) 00000080:00010000:2.0:1775764923.469380:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4b:0x0](ffff9d81843ff448) for lock 0xeef44110eff5773 00000080:00010000:2.0:1775764923.469382:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5773 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.469383:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.469387:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.469400:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.469404:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4b:0x0] 00000008:00010000:2.0:1775764923.469414:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.469416:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce7b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.469440:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.469446:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.469466:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1775764923.470188:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.470197:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.470208:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.470214:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.470221:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.470227:0:10947:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.470234:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.470239:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.470243:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.470248:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.470254:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4b:0x0] updating i_size 0 i_blocks 0 00000100:00100000:1.0:1775764923.470255:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066220416:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.470259:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.470264:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.470267:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce7b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.470270:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.470271:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce7b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.470285:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.470319:0:34105:0:(file.c:2612:do_file_write_iter()) START file setup_kerberos.sh:[0x240004281:0x4b:0x0], ppos: 0, count: 24288 00000080:00200000:2.0:1775764923.470333:0:34105:0:(file.c:2058:ll_file_io_generic()) setup_kerberos.sh: write ppos: 0, bytes: 24288 00000080:00200000:2.0:1775764923.470337:0:34105:0:(file.c:1815:ll_io_set_mirror()) setup_kerberos.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.470341:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000754c054b[0x110, 1, [0x240004281:0x4b:0x0] hash] 00000020:00200000:2.0:1775764923.470343:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 24288) 0 0 00000080:00200000:2.0:1775764923.470346:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4b:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.470352:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 5] 00020000:00200000:2.0:1775764923.470357:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 24288) 24288, 24288 00020000:00200000:2.0:1775764923.470360:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.470373:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 5] 00000008:00010000:2.0:1775764923.470377:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.470379:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce7b0, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.470385:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/550 (0 24575) 00010000:00010000:2.0:1775764923.470397:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.470402:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->24575] (req 0->24575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.470407:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->24575] (req 0->24575) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.470416:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066220544:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.470447:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d49c0 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.471050:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764923.471090:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082330624 00000100:00100000:1.0:1775764923.471094:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082330624 00000100:00100000:1.0:1775764923.471120:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 303 00000100:00100000:1.0:1775764923.471126:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190708700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082330624:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.471133:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.471165:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.471183:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190708700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:31387:x1862023082330624:12345-192.168.202.105@tcp:104: Request processed in 45us (122us total) trans 0 rc 0/0 00010000:00010000:2.0:1775764923.471189:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00000100:00100000:1.0:1775764923.471193:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 303 00010000:00010000:2.0:1775764923.471196:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.471202:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364200 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.471206:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.471230:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.471235:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.471261:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.471266:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.471280:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.471285:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff577a lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155c09 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.471294:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.471295:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.471298:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.471313:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1775764923.471325:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066220672:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.471788:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066220544:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.471797:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->24575] (req 0->24575) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.471804:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->24575] (req 0->24575) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.471809:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->24575] (req 0->24575) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.472046:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082330752 00000100:00100000:3.0:1775764923.472050:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082330752 00000100:00100000:3.0:1775764923.472076:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 304 00000100:00100000:3.0:1775764923.472083:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901ad180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19105:x1862023082330752:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.472092:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->24575] (req 0->24575) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.472101:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->24575] (req 0->24575) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.472110:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.472118:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.472127:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.472136:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.472154:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d8190365a00) 00000100:00100000:3.0:1775764923.472194:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ad180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19105:x1862023082330752:12345-192.168.202.105@tcp:105: Request processed in 110us (184us total) trans 0 rc 0/0 00010000:00010000:2.0:1775764923.472201:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.472201:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 304 00010000:00010000:2.0:1775764923.472215:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.472226:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.472235:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.472245:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.472253:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.472269:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) setup_kerberos.sh: write [0, 24288) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.472278:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.472305:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.472312:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ec6dbc95[3 000000008ded291d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.472315:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003eab9934 17ffffc0001001 3:0 ffff9d819124c7e0 0 lru 00000080:00200000:2.0:1775764923.472322:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a2a56e00 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 0000000068609cb1 > 3< 0 0 > 4< 0 0 8 2887680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.472324:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ec6dbc95 00000080:00200000:2.0:1775764923.472325:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.472327:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.472362:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.472367:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000df74ceac[3 000000008ded291d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.472370:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007348efe0 17ffffc0001001 3:0 ffff9d8186ef0460 1 lru 00000080:00200000:2.0:1775764923.472375:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000766a8339 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 00000000409b8422 0000000068609cb1 > 3< 0 0 > 4< 0 0 8 2887680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.472377:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000df74ceac 00000080:00200000:2.0:1775764923.472378:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.472380:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.472391:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.472394:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000751dd8b8[3 000000008ded291d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.472397:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000349d9c41 17ffffc0001001 3:0 ffff9d8186ef00e0 2 lru 00000080:00200000:2.0:1775764923.472403:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f46ba615 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 00000000409b8422 0000000068609cb1 > 3< 0 0 > 4< 0 0 8 2887680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.472405:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000751dd8b8 00000080:00200000:2.0:1775764923.472406:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.472407:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.472418:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.472422:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003e685bb7[3 000000008ded291d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.472424:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009cfb1c4e 17ffffc0001001 3:0 ffff9d8186ef0b60 3 lru 00000080:00200000:2.0:1775764923.472429:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a99c78ea 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 00000000409b8422 0000000068609cb1 > 3< 0 0 > 4< 0 0 8 2887680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.472431:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003e685bb7 00000080:00200000:2.0:1775764923.472432:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.472434:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.472445:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.472449:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000981e3c2e[3 000000008ded291d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.472451:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b1da51e5 17ffffc0001001 3:0 ffff9d8186ef09a0 4 lru 00000080:00200000:2.0:1775764923.472456:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000062d2c080 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 00000000409b8422 0000000068609cb1 > 3< 0 0 > 4< 0 0 8 2887680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.472458:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000981e3c2e 00000080:00200000:2.0:1775764923.472459:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.472461:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 3808 bytes 00000080:00200000:2.0:1775764923.472481:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 3808, copied 3808 00000080:00200000:2.0:1775764923.472485:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a28cbf29[3 000000008ded291d 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.472487:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cf8bb322 17ffffc0001021 2:0 ffff9d8186ef0540 5 lru 00000080:00200000:2.0:1775764923.472492:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000acdb29d6 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 00000000409b8422 0000000068609cb1 > 3< 0 0 > 4< 0 0 8 2887680 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.472494:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a28cbf29 00000080:00200000:2.0:1775764923.472495:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.472497:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 6, from 0, to 3808 prio 0 00000008:00010000:2.0:1775764923.472508:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 000000006610f5e0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.472520:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000006610f5e0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.472527:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000006610f5e0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.472534:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000006610f5e0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.472540:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000006610f5e0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.472551:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 000000006610f5e0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.472561:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843ff5f8, count 6, dirtied 6 00000080:00200000:2.0:1775764923.472565:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 6 pages 24288 bytes, tot: 24288 00000080:00200000:2.0:1775764923.472567:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) setup_kerberos.sh: write bytes 0, result: 24288 00000020:00010000:2.0:1775764923.472575:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.472578:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.472580:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce7b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.472583:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.472591:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.472600:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.472602:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce7b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.472607:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 24288 bytes 00000080:00200000:2.0:1775764923.472611:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 5] 00000080:00200000:2.0:1775764923.472623:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4b:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.472626:0:34105:0:(file.c:2206:ll_file_io_generic()) setup_kerberos.sh: 2 io complete with rc: 0, result: 24288, restart: 0 00000080:00200000:2.0:1775764923.472631:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4b:0x0] updating i_size 24288 i_blocks 0 00000080:00200000:2.0:1775764923.472634:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 24288 00000080:00200008:2.0:1775764923.472654:0:34105:0:(file.c:447:ll_file_release()) START file setup_kerberos.sh:[0x240004281:0x4b:0x0](ffff9d81843ff448), flags 100001 00000002:00100000:2.0:1775764923.472679:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fce00 x1862023066220288/t38654705907(38654705907) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.472694:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.472699:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fc000 x1862023066220928/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.472713:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066220928:192.168.202.105@tcp:35:cp.0 00000100:00100000:1.0:1775764923.472717:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066220672:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.472758:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d49c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.473461:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fc000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066220928:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.473474:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fce00 x1862023066220288/t38654705907(38654705907) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.473509:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.473518:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.473528:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.473537:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.473544:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.473548:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.473551:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.473556:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.473559:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.473566:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.473580:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.473585:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.473589:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.473592:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.473595:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.473598:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=setup_kerberos.sh, flags=8 00000080:00200008:2.0:1775764923.473603:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file setup_kerberos.sh:[0x240004281:0x4b:0x0](ffff9d81843ff448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.473605:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4b:0x0](ffff9d81843ff448),name=setup_kerberos.sh 00000002:00010000:2.0:1775764923.473612:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4b:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.473616:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.473621:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.473625:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.473627:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4b:0x0](ffff9d81843ff448) 00000080:00010000:2.0:1775764923.473629:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4b:0x0](ffff9d81843ff448) for lock 0xeef44110eff5773 00000080:00010000:2.0:1775764923.473631:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5773 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.473633:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.473637:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.473652:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.473656:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4b:0x0] 00000008:00010000:2.0:1775764923.473665:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.473666:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce7b0, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.473672:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.473678:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.473683:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.473685:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.473690:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000000e9bba4c/0xeef44110eff5781 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c10 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.473694:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.473697:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4b:0x0] updating i_size 24288 i_blocks 0 00000020:00010000:2.0:1775764923.473700:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.473702:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.473704:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce7b0 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.473707:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.473709:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce7b0 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.473721:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.473759:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.473767:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.473774:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.473782:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.473788:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.473792:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.473794:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.473798:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.473801:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.473807:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.473815:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.473818:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.473821:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.473824:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.473826:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.473833:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup-nfs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.473837:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-nfs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.473844:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup-nfs.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.473866:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.473872:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.473896:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.473906:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066221056:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.473933:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d49c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.474711:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066221056:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.474718:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474723:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474727:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474731:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474734:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474737:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190365600 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.474740:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474743:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474747:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474750:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.474753:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474757:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff5788 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.474763:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fca80 x1862023066221056/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.474768:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.474776:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.474793:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.474798:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.474801:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.474807:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.474811:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.474812:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.474814:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.474817:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.474820:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.474827:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.474835:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.474839:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.474842:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.474845:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.474847:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.474854:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=setup-nfs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.474857:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-nfs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.474864:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup-nfs.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.474895:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474900:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.474904:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.474911:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066221184:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.474937:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d49c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.475729:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066221184:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.475735:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475740:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475744:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475747:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475754:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475758:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190365c00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.475759:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475763:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475766:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475770:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.475772:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475775:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365c00/0xeef44110eff578f lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.475781:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d81906fca80 x1862023066221184/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.475787:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.475795:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.475826:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.475831:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.475835:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.475844:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.475848:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.475850:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.475852:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.475854:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.475856:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.475860:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.475865:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.475868:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.475870:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.475871:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.475873:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.475895:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.475900:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=setup-nfs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da180, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.475906:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=setup-nfs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.475918:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: setup-nfs.sh,[0x240004281:0x4c:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.475942:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.475950:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.475956:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.475959:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d81906fca80 x1862023066221440/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.475964:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.475972:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066221440:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.476047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d49c0 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.477062:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81906fca80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066221440:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.477070:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477079:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477084:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477090:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477095:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477100:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477107:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477111:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.477116:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d81906fca80 x1862023066221440/t38654705909(38654705909) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.477123:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fca80 x1862023066221440/t38654705909(38654705909) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.477129:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.477135:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477139:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/76 (0 0) 00000080:00200000:2.0:1775764923.477142:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.477146:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=setup-nfs.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.477168:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x4c:0x0](ffff9d81843fe3c8) 00000080:00200000:2.0:1775764923.477231:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x4c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.477237:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x4c:0x0](ffff9d81843fe3c8) 00000080:00010000:2.0:1775764923.477239:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4c:0x0](ffff9d81843fe3c8) for lock 0xeef44110eff5796 00000080:00010000:2.0:1775764923.477243:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4c:0x0](ffff9d81843fe3c8) 00000080:00200008:2.0:1775764923.477248:0:34105:0:(file.c:1057:ll_file_open()) START file setup-nfs.sh:[0x240004281:0x4c:0x0](ffff9d81843fe3c8), flags 100301 00000002:00100000:2.0:1775764923.477257:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d81906fca80 x1862023066221440/t38654705909(38654705909) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.477278:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5796 from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.477281:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477286:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.477294:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x4c:0x0](ffff9d81843fe3c8), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.477309:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file setup-nfs.sh:[0x240004281:0x4c:0x0](ffff9d81843fe3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.477311:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4c:0x0](ffff9d81843fe3c8),name=setup-nfs.sh 00000002:00010000:2.0:1775764923.477318:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4c:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.477323:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.477328:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.477332:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.477334:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4c:0x0](ffff9d81843fe3c8) 00000080:00010000:2.0:1775764923.477336:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4c:0x0](ffff9d81843fe3c8) for lock 0xeef44110eff5796 00000080:00010000:2.0:1775764923.477338:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5796 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.477340:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.477344:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.477356:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.477361:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4c:0x0] 00000008:00010000:2.0:1775764923.477372:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.477373:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cea30, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.477391:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.477399:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.477425:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.477446:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066221568:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.478189:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.478199:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.478206:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.478215:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.478222:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.478228:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.478236:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.478241:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.478245:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.478250:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.478256:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066221568:192.168.202.105@tcp:101:cp.0 00000080:00200000:2.0:1775764923.478267:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4c:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.478271:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.478274:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.478276:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cea30 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.478279:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.478280:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cea30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.478309:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.478361:0:34105:0:(file.c:2612:do_file_write_iter()) START file setup-nfs.sh:[0x240004281:0x4c:0x0], ppos: 0, count: 1611 00000080:00200000:2.0:1775764923.478369:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 1611 bytes 00000080:00200000:2.0:1775764923.478373:0:34105:0:(file.c:2588:ll_do_tiny_write()) result: 0, original count 1611 00000080:00200000:2.0:1775764923.478383:0:34105:0:(file.c:2058:ll_file_io_generic()) setup-nfs.sh: write ppos: 0, bytes: 1611 00000080:00200000:2.0:1775764923.478385:0:34105:0:(file.c:1815:ll_io_set_mirror()) setup-nfs.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.478389:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@00000000a8b2cb79[0x110, 1, [0x240004281:0x4c:0x0] hash] 00000020:00200000:2.0:1775764923.478390:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1611) 0 0 00000080:00200000:2.0:1775764923.478392:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4c:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.478396:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1775764923.478400:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1611) 1611, 1611 00020000:00200000:2.0:1775764923.478403:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.478413:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1775764923.478418:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.478420:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cea30, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.478425:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/551 (0 4095) 00010000:00010000:2.0:1775764923.478438:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.478444:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.478449:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 3/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.478457:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066221696:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.478482:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4900 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.479248:0:34090:0:(service.c:153:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1775764923.479256:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082330880 00000100:00100000:1.0:1775764923.479259:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082330880 00000100:00100000:1.0:1775764923.479286:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 305 00000100:00100000:1.0:1775764923.479291:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d818796e850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082330880:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.479299:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.479348:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.479354:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d818796e850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25528:x1862023082330880:12345-192.168.202.105@tcp:104: Request processed in 63us (150us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.479360:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 305 00010000:00010000:2.0:1775764923.479367:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.479374:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.479384:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364a00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.479389:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.479433:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.479439:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.479477:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 2/0,0 mode: --/PR res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.479487:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.479503:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.479511:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000c2d52f94/0xeef44110eff579d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155c64 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.479524:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066221824:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.479525:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.479528:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.479536:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.479907:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066221696:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.479916:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.479922:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.479927:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.480332:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082331008 00000100:00100000:3.0:1775764923.480336:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082331008 00000100:00100000:3.0:1775764923.480356:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 306 00000100:00100000:3.0:1775764923.480361:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901ad500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19106:x1862023082331008:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.480367:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.480377:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.480386:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 5/0,1 mode: --/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.480394:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.480408:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.480417:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.480435:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d8190364e00) 00010000:00010000:2.0:1775764923.480452:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.480459:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ad500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19106:x1862023082331008:12345-192.168.202.105@tcp:105: Request processed in 96us (164us total) trans 0 rc 0/0 00010000:00010000:2.0:1775764923.480462:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.480468:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 306 00010000:00010000:2.0:1775764923.480481:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.480488:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.480494:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.480498:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.480512:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) setup-nfs.sh: write [0, 1611) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.480519:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 1611 bytes 00000080:00200000:2.0:1775764923.480541:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 1611, copied 1611 00000080:00200000:2.0:1775764923.480546:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b9e4c33a[3 000000000f25c0f5 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.480548:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000052c98817 17ffffc0001001 3:0 ffff9d8186ef0000 0 lru 00000080:00200000:2.0:1775764923.480553:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009a02248c 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 00000000bf8b6859 > 3< 0 0 > 4< 0 0 8 196608 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.480555:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b9e4c33a 00000080:00200000:2.0:1775764923.480555:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.480558:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 1, from 0, to 1611 prio 0 00000008:00010000:2.0:1775764923.480567:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000e3c8cd38 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.480576:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fe578, count 1, dirtied 1 00000080:00200000:2.0:1775764923.480579:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 1 pages 1611 bytes, tot: 1611 00000080:00200000:2.0:1775764923.480581:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) setup-nfs.sh: write bytes 0, result: 1611 00000020:00010000:2.0:1775764923.480589:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.480591:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.480593:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cea30 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.480596:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.480601:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.480607:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.480609:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cea30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.480612:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 1611 bytes 00000080:00200000:2.0:1775764923.480614:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1775764923.480632:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4c:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.480634:0:34105:0:(file.c:2206:ll_file_io_generic()) setup-nfs.sh: 2 io complete with rc: 0, result: 1611, restart: 0 00000080:00200000:2.0:1775764923.480638:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4c:0x0] updating i_size 1611 i_blocks 0 00000080:00200000:2.0:1775764923.480640:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 1611 00000080:00200008:2.0:1775764923.480660:0:34105:0:(file.c:447:ll_file_release()) START file setup-nfs.sh:[0x240004281:0x4c:0x0](ffff9d81843fe3c8), flags 100001 00000002:00100000:2.0:1775764923.480680:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d81906fca80 x1862023066221440/t38654705909(38654705909) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.480687:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.480690:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190513b80 x1862023066222080/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.480698:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190513b80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222080:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.480734:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4900 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.481005:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066221824:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.481489:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190513b80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222080:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.481504:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d81906fca80 x1862023066221440/t38654705909(38654705909) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.481534:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.481541:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.481548:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.481555:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.481559:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.481562:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.481564:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.481566:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.481568:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.481573:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.481578:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.481581:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.481583:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.481585:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.481587:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.481588:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=setup-nfs.sh, flags=8 00000080:00200008:2.0:1775764923.481592:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file setup-nfs.sh:[0x240004281:0x4c:0x0](ffff9d81843fe3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.481594:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4c:0x0](ffff9d81843fe3c8),name=setup-nfs.sh 00000002:00010000:2.0:1775764923.481599:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4c:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.481603:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.481607:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.481611:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.481613:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4c:0x0](ffff9d81843fe3c8) 00000080:00010000:2.0:1775764923.481614:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4c:0x0](ffff9d81843fe3c8) for lock 0xeef44110eff5796 00000080:00010000:2.0:1775764923.481616:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff5796 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.481618:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.481622:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.481637:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.481641:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4c:0x0] 00000008:00010000:2.0:1775764923.481650:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.481651:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876cea30, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.481657:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.481663:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 3/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.481668:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.481671:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.481675:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000edaf1b27/0xeef44110eff57a4 lrc: 5/0,0 mode: PW/PW res: [0x280000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155c6b expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.481680:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.481683:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4c:0x0] updating i_size 1611 i_blocks 0 00000020:00010000:2.0:1775764923.481685:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.481686:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.481688:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876cea30 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.481689:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.481691:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876cea30 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.481699:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.481723:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.481728:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.481732:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.481737:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.481741:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.481743:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.481745:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.481747:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.481749:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.481756:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.481761:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.481763:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.481764:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.481766:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.481767:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.481774:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sgpdd-survey.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.481776:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sgpdd-survey.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.481781:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sgpdd-survey.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.481795:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.481799:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.481804:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.481811:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222208:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.481843:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.482729:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190510700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222208:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.482739:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482748:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482754:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482760:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482766:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482771:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364800 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.482774:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482780:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482786:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482806:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.482811:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482816:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364800/0xeef44110eff57ab lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.482828:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d8190510700 x1862023066222208/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.482837:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.482848:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.482872:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.482895:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.482900:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.482906:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.482910:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.482913:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.482915:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.482917:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.482919:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.482923:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.482928:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.482931:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.482933:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.482935:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.482936:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.482941:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=sgpdd-survey.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.482943:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sgpdd-survey.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.482947:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sgpdd-survey.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.482960:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482967:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.482971:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.482980:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222336:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.483019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.483796:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190510700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222336:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.483803:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483808:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483812:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483815:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483819:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483822:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364000 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.483823:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483826:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483830:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483834:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.483836:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.483839:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364000/0xeef44110eff57b2 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.483846:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d8190510700 x1862023066222336/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.483851:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.483858:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.483911:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.483919:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.483925:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.483933:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.483937:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.483940:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.483941:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.483944:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.483945:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.483954:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.483959:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.483961:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.483963:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.483965:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.483966:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.483974:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.483977:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=sgpdd-survey.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851dac00, open_flags 80c1, mode 81a4 opened 0 00000080:00200000:2.0:1775764923.483981:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=sgpdd-survey.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.483989:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: sgpdd-survey.sh,[0x240004281:0x4d:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.484006:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.484010:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.484015:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.484018:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190510700 x1862023066222592/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.484023:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.484038:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222592:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.484098:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4900 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.485111:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190510700 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222592:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.485119:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485127:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485138:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485144:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485149:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485155:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485159:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485163:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.485168:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d8190510700 x1862023066222592/t38654705911(38654705911) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.485187:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d8190510700 x1862023066222592/t38654705911(38654705911) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 656/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.485203:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.485212:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485220:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/77 (0 0) 00000080:00200000:2.0:1775764923.485224:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.485230:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=sgpdd-survey.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.485255:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x4d:0x0](ffff9d81843fb248) 00000080:00200000:2.0:1775764923.485323:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x4d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.485333:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x4d:0x0](ffff9d81843fb248) 00000080:00010000:2.0:1775764923.485336:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4d:0x0](ffff9d81843fb248) for lock 0xeef44110eff57b9 00000080:00010000:2.0:1775764923.485342:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4d:0x0](ffff9d81843fb248) 00000080:00200008:2.0:1775764923.485349:0:34105:0:(file.c:1057:ll_file_open()) START file sgpdd-survey.sh:[0x240004281:0x4d:0x0](ffff9d81843fb248), flags 100301 00000002:00100000:2.0:1775764923.485362:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d8190510700 x1862023066222592/t38654705911(38654705911) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.485386:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff57b9 from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.485390:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485397:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.485409:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x4d:0x0](ffff9d81843fb248), inode mode 81a4 mask 40 00000080:00200008:2.0:1775764923.485420:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sgpdd-survey.sh:[0x240004281:0x4d:0x0](ffff9d81843fb248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.485424:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4d:0x0](ffff9d81843fb248),name=sgpdd-survey.sh 00000002:00010000:2.0:1775764923.485433:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4d:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.485440:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.485445:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.485449:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.485452:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4d:0x0](ffff9d81843fb248) 00000080:00010000:2.0:1775764923.485454:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4d:0x0](ffff9d81843fb248) for lock 0xeef44110eff57b9 00000080:00010000:2.0:1775764923.485456:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff57b9 from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.485458:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.485462:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.485476:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.485481:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4d:0x0] 00000008:00010000:2.0:1775764923.485494:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.485496:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce030, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.485520:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.485529:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 3/1,0 mode: --/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.485558:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1775764923.486315:0:10947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.486325:0:10947:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 6/2,0 mode: --/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.486336:0:10947:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.486345:0:10947:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.486355:0:10947:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 5/2,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.486371:0:10947:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.486381:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 6/2,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.486388:0:10947:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.486393:0:10947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 5/1,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.486398:0:10947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 4/0,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.486406:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4d:0x0] updating i_size 0 i_blocks 0 00000100:00100000:1.0:1775764923.486407:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190512300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066222720:192.168.202.105@tcp:101:cp.0 00000020:00010000:2.0:1775764923.486414:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.486418:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.486421:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce030 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.486425:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.486428:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce030 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.486443:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.486465:0:34105:0:(file.c:2612:do_file_write_iter()) START file sgpdd-survey.sh:[0x240004281:0x4d:0x0], ppos: 0, count: 1765 00000080:00200000:2.0:1775764923.486478:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 1765 bytes 00000080:00200000:2.0:1775764923.486480:0:34105:0:(file.c:2588:ll_do_tiny_write()) result: 0, original count 1765 00000080:00200000:2.0:1775764923.486490:0:34105:0:(file.c:2058:ll_file_io_generic()) sgpdd-survey.sh: write ppos: 0, bytes: 1765 00000080:00200000:2.0:1775764923.486493:0:34105:0:(file.c:1815:ll_io_set_mirror()) sgpdd-survey.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.486497:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@000000007ef64a67[0x110, 1, [0x240004281:0x4d:0x0] hash] 00000020:00200000:2.0:1775764923.486499:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 1765) 0 0 00000080:00200000:2.0:1775764923.486502:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4d:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.486507:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:2.0:1775764923.486512:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 1765) 1765, 1765 00020000:00200000:2.0:1775764923.486515:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.486530:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:2.0:1775764923.486537:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.486540:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce030, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.486547:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8190486c00 type 11 mode 2 res 11811161088/551 (0 4095) 00010000:00010000:2.0:1775764923.486560:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.486565:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.486574:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.486582:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222848:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.486607:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4f00 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.487325:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764923.487368:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082331136 00000100:00100000:1.0:1775764923.487373:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082331136 00000100:00100000:1.0:1775764923.487399:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 307 00000100:00100000:1.0:1775764923.487406:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d8190709c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082331136:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.487414:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.487444:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.487449:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d8190709c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:25525:x1862023082331136:12345-192.168.202.105@tcp:104: Request processed in 44us (125us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.487454:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 307 00010000:00010000:2.0:1775764923.487466:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.487472:0:33224:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 2/0,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.487480:0:33224:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583aa00 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.487483:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 3/0,0 mode: PR/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.487510:0:33224:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.487515:0:33224:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.487541:0:33224:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 2/0,0 mode: --/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.487546:0:33224:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.487561:0:33224:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 1/0,0 mode: --/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.487566:0:33224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 00000000b3465b79/0xeef44110eff57c0 lrc: 0/0,0 mode: --/PR res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155cbf expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.487584:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.487587:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.487590:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764923.487591:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066222976:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.487996:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190510000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066222848:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.488009:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.488018:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.488026:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.488350:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082331264 00000100:00100000:3.0:1775764923.488355:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082331264 00000100:00100000:3.0:1775764923.488384:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 308 00000100:00100000:3.0:1775764923.488389:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901ad880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:22026:x1862023082331264:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.488395:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.488401:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.488407:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 5/0,1 mode: --/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.488416:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.488421:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.488427:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.488440:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d818583a000) 00010000:00010000:2.0:1775764923.488455:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.488463:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901ad880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:22026:x1862023082331264:12345-192.168.202.105@tcp:105: Request processed in 73us (168us total) trans 0 rc 0/0 00010000:00010000:2.0:1775764923.488468:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.488468:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 308 00010000:00010000:2.0:1775764923.488477:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.488484:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,2 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.488494:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 5/0,2 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.488499:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.488512:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) sgpdd-survey.sh: write [0, 1765) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.488518:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 1765 bytes 00000080:00200000:2.0:1775764923.488543:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 1765, copied 1765 00000080:00200000:2.0:1775764923.488548:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005527d355[3 000000009c57cbf8 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.488550:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002e8ed052 17ffffc0001001 3:0 ffff9d8186ef02a0 0 lru 00000080:00200000:2.0:1775764923.488554:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000de413da4 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 00000000409b8422 000000004e997501 > 3< 0 0 > 4< 0 0 8 2838528 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.488556:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005527d355 00000080:00200000:2.0:1775764923.488557:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.488559:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 1, from 0, to 1765 prio 0 00000008:00010000:2.0:1775764923.488568:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 000000002b6030ac ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.488578:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fb3f8, count 1, dirtied 1 00000080:00200000:2.0:1775764923.488582:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 1 pages 1765 bytes, tot: 1765 00000080:00200000:2.0:1775764923.488585:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) sgpdd-survey.sh: write bytes 0, result: 1765 00000020:00010000:2.0:1775764923.488594:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.488597:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.488600:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce030 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.488604:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 5/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.488617:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.488628:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.488630:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce030 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.488636:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 1765 bytes 00000080:00200000:2.0:1775764923.488640:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:2.0:1775764923.488654:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.488657:0:34105:0:(file.c:2206:ll_file_io_generic()) sgpdd-survey.sh: 2 io complete with rc: 0, result: 1765, restart: 0 00000080:00200000:2.0:1775764923.488663:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4d:0x0] updating i_size 1765 i_blocks 0 00000080:00200000:2.0:1775764923.488665:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 1765 00000080:00200008:2.0:1775764923.488686:0:34105:0:(file.c:447:ll_file_release()) START file sgpdd-survey.sh:[0x240004281:0x4d:0x0](ffff9d81843fb248), flags 100001 00000002:00100000:2.0:1775764923.488715:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d8190510700 x1862023066222592/t38654705911(38654705911) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.488724:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.488729:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190510000 x1862023066223232/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.488743:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066223232:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.488780:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4f00 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.489007:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066222976:192.168.202.105@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1775764923.489587:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190510000 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066223232:192.168.202.105@tcp:35:cp.0 00000002:00100000:2.0:1775764923.489603:0:34105:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9d8190510700 x1862023066222592/t38654705911(38654705911) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 584/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.489641:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.489651:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.489662:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.489671:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.489679:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.489683:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.489686:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.489690:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.489693:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.489698:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.489704:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.489707:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.489710:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.489712:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.489714:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.489716:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=sgpdd-survey.sh, flags=8 00000080:00200008:2.0:1775764923.489719:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file sgpdd-survey.sh:[0x240004281:0x4d:0x0](ffff9d81843fb248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.489722:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4d:0x0](ffff9d81843fb248),name=sgpdd-survey.sh 00000002:00010000:2.0:1775764923.489729:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4d:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.489735:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.489748:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.489754:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.489758:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4d:0x0](ffff9d81843fb248) 00000080:00010000:2.0:1775764923.489761:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4d:0x0](ffff9d81843fb248) for lock 0xeef44110eff57b9 00000080:00010000:2.0:1775764923.489765:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff57b9 from it ffffbc1987ddfcb8 00010000:00010000:2.0:1775764923.489768:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.489775:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775764923.489799:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.489806:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4d:0x0] 00000008:00010000:2.0:1775764923.489821:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.489824:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce030, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.489833:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.489843:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1775764923.489852:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1775764923.489855:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 6/0,1 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.489868:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 5/0,0 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1775764923.489891:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1775764923.489896:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4d:0x0] updating i_size 1765 i_blocks 0 00000020:00010000:2.0:1775764923.489899:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.489901:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.489904:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce030 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.489907:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.489909:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce030 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.489923:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.489963:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.489972:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.489979:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.489992:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.489999:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.490003:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.490005:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.490008:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.490011:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.490023:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.490032:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.490035:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.490039:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.490042:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.490045:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.490053:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=test-framework.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=9 00000080:00200000:2.0:1775764923.490057:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=test-framework.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.490065:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: test-framework.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.490088:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.490095:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.490101:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.490110:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190511c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066223360:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.490142:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.491011:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190511c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066223360:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.491018:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491023:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491027:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491030:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491034:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491037:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583be00 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.491039:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491043:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491047:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491050:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.491053:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491057:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583be00/0xeef44110eff57ce lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.491064:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d8190511c00 x1862023066223360/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.491070:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.491079:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.491098:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.491102:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.491106:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.491111:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.491115:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.491117:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.491119:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.491121:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfb58 00010000:00010000:2.0:1775764923.491123:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.491127:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.491132:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.491134:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=24 00000080:00200000:2.0:1775764923.491136:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.491138:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=24 00000080:00200000:2.0:1775764923.491139:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.491146:0:34105:0:(namei.c:1322:ll_lookup_nd()) VFS Op:name=test-framework.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), flags=8 00000080:00200000:2.0:1775764923.491148:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=test-framework.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=getattr 00000002:00010000:2.0:1775764923.491153:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: test-framework.sh,[0x0:0x0:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.491164:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491186:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.491191:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 3/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.491199:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190511c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066223488:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.491230:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.491996:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190511c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066223488:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.492003:0:34105:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492008:0:34105:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492011:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 4/1,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492015:0:34105:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492018:0:34105:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492021:0:34105:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d818583b600 already unused, calling callback (ffffffffc1c0c2e0) 00010000:00010000:2.0:1775764923.492023:0:34105:0:(ldlm_request.c:1724:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492026:0:34105:0:(ldlm_request.c:1734:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 4/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492030:0:34105:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 5/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492034:0:34105:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1775764923.492036:0:34105:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 3/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492039:0:34105:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b600/0xeef44110eff57d5 lrc: 0/0,0 mode: --/CR res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.492046:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@ffff9d8190511c00 x1862023066223488/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/584 e 0 to 0 dl 1775764939 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'cp.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1775764923.492051:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1775764923.492058:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 30 res 9663693441/1 (0 0) 00000080:00200000:2.0:1775764923.492093:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448),name=/ 00000002:00010000:2.0:1775764923.492098:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.492102:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.492108:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.492111:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.492114:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) 00000080:00010000:2.0:1775764923.492115:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9d81842cf448) for lock 0xeef44110eff4226 00000080:00010000:2.0:1775764923.492117:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff4226 from it ffffbc1987ddfbf8 00010000:00010000:2.0:1775764923.492119:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1775764923.492127:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d81892c2000 lock: ffff9d8190390e00/0xeef44110eff4226 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa3215238b expref: -99 pid: 33918 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1775764923.492132:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9d81842cf448), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.492134:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d4b.sanity-scrub, flags=25 00000080:00200000:2.0:1775764923.492137:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x200004a52:0x1:0x0](ffff9d818436db88), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.492138:0:34105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=mds2, flags=25 00000080:00200000:2.0:1775764923.492139:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 1 00000080:00200000:2.0:1775764923.492144:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x1:0x0](ffff9d818436d348), inode mode 41ed mask 3 00000080:00200000:2.0:1775764923.492147:0:34105:0:(namei.c:1399:ll_atomic_open()) VFS Op:name=test-framework.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), file ffff9d81851da000, open_flags 80c1, mode 81ed opened 0 00000080:00200000:2.0:1775764923.492151:0:34105:0:(namei.c:1063:ll_lookup_it()) VFS Op:name=test-framework.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000002:00010000:2.0:1775764923.492159:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: test-framework.sh,[0x240004281:0x4e:0x0]) in obj [0x240004281:0x1:0x0], intent: open|creat flags 0100302 00010000:00010000:2.0:1775764923.492189:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.492194:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1775764923.492199:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1775764923.492202:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190511c00 x1862023066223744/t0(0) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1775764923.492207:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 3/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 34105 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.492215:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190511c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066223744:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.492287:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4f00 going to sleep for 11 seconds 00000100:00100000:2.0:1775764923.493465:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190511c00 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066223744:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.493474:0:34105:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 4/0,1 mode: --/CW res: [0x240004281:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493483:0:34105:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493487:0:34105:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 4/0,1 mode: --/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493494:0:34105:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493499:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 4/0,1 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493504:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493508:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 5/1,1 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493513:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 4/1,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1775764923.493518:0:34105:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=3 disposition=37, status=0 req@ffff9d8190511c00 x1862023066223744/t38654705913(38654705913) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1775764923.493525:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d8190511c00 x1862023066223744/t38654705913(38654705913) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 664/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1775764923.493531:0:34105:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1775764923.493538:0:34105:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493543:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d8186cee000 type 13 mode 4 res 9663693441/78 (0 0) 00000080:00200000:2.0:1775764923.493547:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 3 on i1 [0x240004281:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1775764923.493553:0:34105:0:(namei.c:1610:ll_create_it()) VFS Op:name=test-framework.sh, dir=[0x240004281:0x1:0x0](ffff9d818436d348), intent=open|creat 00000080:00200000:2.0:1775764923.493572:0:34105:0:(llite_lib.c:3177:ll_read_inode2()) VFS Op:inode=[0x240004281:0x4e:0x0](ffff9d81843fc2c8) 00000080:00200000:2.0:1775764923.493647:0:34105:0:(file.c:6766:ll_layout_conf()) [0x240004281:0x4e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1775764923.493653:0:34105:0:(namei.c:1576:ll_create_node()) setting l_ast_data to inode [0x240004281:0x4e:0x0](ffff9d81843fc2c8) 00000080:00010000:2.0:1775764923.493655:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4e:0x0](ffff9d81843fc2c8) for lock 0xeef44110eff57dc 00000080:00010000:2.0:1775764923.493659:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4e:0x0](ffff9d81843fc2c8) 00000080:00200008:2.0:1775764923.493665:0:34105:0:(file.c:1057:ll_file_open()) START file test-framework.sh:[0x240004281:0x4e:0x0](ffff9d81843fc2c8), flags 100301 00000002:00100000:2.0:1775764923.493675:0:34105:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9d8190511c00 x1862023066223744/t38654705913(38654705913) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'cp.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1775764923.493698:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff57dc from it ffff9d8184e989c0 00010000:00010000:2.0:1775764923.493700:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493706:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.493715:0:34105:0:(file.c:6593:ll_inode_permission()) VFS Op:inode=[0x240004281:0x4e:0x0](ffff9d81843fc2c8), inode mode 81ed mask 40 00000080:00200008:2.0:1775764923.493725:0:34105:0:(file.c:6226:ll_getattr_dentry()) START file test-framework.sh:[0x240004281:0x4e:0x0](ffff9d81843fc2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1775764923.493728:0:34105:0:(file.c:6121:ll_inode_revalidate()) VFS Op:inode=[0x240004281:0x4e:0x0](ffff9d81843fc2c8),name=test-framework.sh 00000002:00010000:2.0:1775764923.493735:0:34105:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x240004281:0x4e:0x0]) in obj [0x240004281:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1775764923.493739:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.493745:0:34105:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 2/1,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.493748:0:34105:0:(namei.c:943:ll_intent_lock()) intent lock 8 on i1 [0x240004281:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1775764923.493751:0:34105:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240004281:0x4e:0x0](ffff9d81843fc2c8) 00000080:00010000:2.0:1775764923.493752:0:34105:0:(llite_internal.h:1984:ll_set_lock_data()) setting l_data to inode [0x240004281:0x4e:0x0](ffff9d81843fc2c8) for lock 0xeef44110eff57dc 00000080:00010000:2.0:1775764923.493754:0:34105:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0xeef44110eff57dc from it ffffbc1987ddfcd8 00010000:00010000:2.0:1775764923.493756:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 3/1,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.493760:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775764923.493773:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1775764923.493779:0:34105:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x240004281:0x4e:0x0] 00000008:00010000:2.0:1775764923.493795:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1775764923.493798:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce130, osc lock ffff9d8198380980, flags 0x400001002 00010000:00010000:2.0:1775764923.493824:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.493834:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1775764923.493868:0:34105:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1775764923.493903:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066223872:192.168.202.105@tcp:101:cp.0 00010000:00010000:1.0:1775764923.494667:0:10945:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.494676:0:10945:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.494684:0:10945:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.494691:0:10945:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1775764923.494698:0:10945:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1775764923.494703:0:10945:0:(cl_io.c:1822:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1775764923.494710:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.494716:0:10945:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.494720:0:10945:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1775764923.494726:0:10945:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1775764923.494734:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066223872:192.168.202.105@tcp:101:cp.0 00000080:00200000:2.0:1775764923.494737:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4e:0x0] updating i_size 0 i_blocks 0 00000020:00010000:2.0:1775764923.494742:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.494745:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.494748:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce130 (ffff9d8191255758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.494752:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81887356d0 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.494755:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce130 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.494772:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1775764923.494985:0:34105:0:(file.c:2612:do_file_write_iter()) START file test-framework.sh:[0x240004281:0x4e:0x0], ppos: 0, count: 313063 00000080:00200000:2.0:1775764923.495002:0:34105:0:(file.c:2058:ll_file_io_generic()) test-framework.sh: write ppos: 0, bytes: 313063 00000080:00200000:2.0:1775764923.495007:0:34105:0:(file.c:1815:ll_io_set_mirror()) test-framework.sh: desiginated mirror: 0 00000020:00200000:2.0:1775764923.495012:0:34105:0:(cl_io.c:214:cl_io_rw_init()) header@000000008a5acd64[0x110, 1, [0x240004281:0x4e:0x0] hash] 00000020:00200000:2.0:1775764923.495014:0:34105:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [0, 313063) 0 0 00000080:00200000:2.0:1775764923.495017:0:34105:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4e:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.495029:0:34105:0:(file.c:2136:ll_file_io_generic()) Range lock [0, 76] 00020000:00200000:2.0:1775764923.495035:0:34105:0:(lov_io.c:1052:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 313063) 313063, 313063 00020000:00200000:2.0:1775764923.495039:0:34105:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1775764923.495057:0:34105:0:(vvp_io.c:198:vvp_io_one_lock_index()) lock: 1 [0, 76] 00000008:00010000:2.0:1775764923.495063:0:34105:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1775764923.495065:0:34105:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9d81876ce130, osc lock ffff9d8198380980, flags 0x0 00010000:00010000:2.0:1775764923.495071:0:34105:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9d819abf5800 type 11 mode 2 res 10737419264/552 (0 315391) 00010000:00010000:2.0:1775764923.495097:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 3/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.495107:0:34105:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 3/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->315391] (req 0->315391) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.495113:0:34105:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 3/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->315391] (req 0->315391) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1775764923.495123:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190512680 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066224000:192.168.202.105@tcp:101:cp.0 00000100:00100000:2.0:1775764923.495150:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4d80 going to sleep for 11 seconds 00000100:00100000:3.0:1775764923.495895:0:10939:0:(events.c:340:request_in_callback()) peer: 12345-192.168.202.105@tcp (source: 12345-192.168.202.105@tcp) 00000100:00100000:1.0:1775764923.495944:0:34090:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082331392 00000100:00100000:1.0:1775764923.495949:0:34090:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082331392 00000100:00100000:1.0:1775764923.495993:0:34090:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 309 00000100:00100000:1.0:1775764923.496002:0:34090:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d819070aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082331392:12345-192.168.202.105@tcp:104: 00010000:00010000:1.0:1775764923.496011:0:34090:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1775764923.496058:0:34090:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1775764923.496065:0:34090:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d819070aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:LOV_OSC_UUID+4:34325:x1862023082331392:12345-192.168.202.105@tcp:104: Request processed in 64us (191us total) trans 0 rc 0/0 00000100:00100000:1.0:1775764923.496072:0:34090:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 309 00010000:00010000:2.0:1775764923.496082:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775764923.496090:0:33225:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.496101:0:33225:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9d8190364200 already unused, calling callback (ffffffffc12b72c0) 00010000:00010000:2.0:1775764923.496107:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400000000 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1775764923.496143:0:33225:0:(vvp_io.c:1817:vvp_io_init()) [0x240004281:0x4e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1775764923.496151:0:33225:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1775764923.496208:0:33225:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.496217:0:33225:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1775764923.496240:0:33225:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.496248:0:33225:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000017556004/0xeef44110eff57e3 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x228:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400000020 nid: local remote: 0x4dcde3aa32155d1a expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775764923.496262:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764923.496265:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764923.496266:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190511f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066224128:192.168.202.105@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1775764923.496269:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764923.496739:0:34105:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190512680 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066224000:192.168.202.105@tcp:101:cp.0 00010000:00010000:2.0:1775764923.496749:0:34105:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->315391] (req 0->315391) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.496756:0:34105:0:(ldlm_request.c:298:ldlm_completion_ast()) ### client-side enqueue returned a blocked locksleeping ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->315391] (req 0->315391) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.496761:0:34105:0:(ldlm_request.c:327:ldlm_completion_ast()) ### waiting indefinitely because of NO_TIMEOUT ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->315391] (req 0->315391) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.497116:0:33219:0:(service.c:2280:ptlrpc_server_handle_req_in()) unwrap req x1862023082331520 00000100:00100000:3.0:1775764923.497120:0:33219:0:(service.c:2345:ptlrpc_server_handle_req_in()) got req x1862023082331520 00000100:00100000:3.0:1775764923.497142:0:33219:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.105@tcp, seq: 310 00000100:00100000:3.0:1775764923.497147:0:33219:0:(service.c:2531:ptlrpc_server_handle_request()) Handling RPC req@ffff9d81901aea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19104:x1862023082331520:12345-192.168.202.105@tcp:105: 00010000:00010000:3.0:1775764923.497152:0:33219:0:(ldlm_lockd.c:2518:ldlm_callback_handler()) ### completion ast ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->315391] (req 0->315391) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.497159:0:33219:0:(ldlm_lockd.c:1968:ldlm_handle_cp_callback()) ### client completion callback handler START ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->315391] (req 0->315391) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.497183:0:33219:0:(ldlm_lockd.c:2045:ldlm_handle_cp_callback()) ### completion AST, new policy data ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: --/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.497192:0:33219:0:(ldlm_lock.c:715:ldlm_add_cp_work_item()) ### lock granted; sending completion AST. ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.497201:0:33219:0:(ldlm_resource.c:1714:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x10080000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.497209:0:33219:0:(ldlm_lockd.c:2073:ldlm_handle_cp_callback()) ### callback handler finished, about to run_ast_work ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 6/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x80000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1775764923.497230:0:33219:0:(ldlm_lockd.c:2084:ldlm_handle_cp_callback()) ### client completion callback handler END (lock ffff9d8190010800) 00010000:00010000:2.0:1775764923.497256:0:34105:0:(ldlm_request.c:193:ldlm_completion_tail()) ### client-side enqueue: granted ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.497260:0:33219:0:(service.c:2589:ptlrpc_server_handle_request()) Handled RPC req@ffff9d81901aea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:19104:x1862023082331520:12345-192.168.202.105@tcp:105: Request processed in 110us (188us total) trans 0 rc 0/0 00010000:00010000:2.0:1775764923.497274:0:34105:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x20000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1775764923.497275:0:33219:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.105@tcp, seq: 310 00010000:00010000:2.0:1775764923.497290:0:34105:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x10000000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.497298:0:34105:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,2 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x10020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.497305:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,2 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.497310:0:34105:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.497328:0:34105:0:(vvp_io.c:1286:vvp_io_write_start()) test-framework.sh: write [0, 313063) append 0 append lockpos 0 00000080:00200000:2.0:1775764923.497341:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 0 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497375:0:34105:0:(rw26.c:858:ll_write_end()) pos 0, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497382:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ab73d6ac[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497385:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000064b4381d 17ffffc0001001 3:0 ffff9d8186ef0ee0 0 lru 00000080:00200000:2.0:1775764923.497391:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000005d96ff69 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497393:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ab73d6ac 00000080:00200000:2.0:1775764923.497394:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 1. 00000080:00200000:2.0:1775764923.497396:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 1 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497411:0:34105:0:(rw26.c:858:ll_write_end()) pos 4096, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497413:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008457470c[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497415:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000cfa1749d 17ffffc0001001 3:0 ffff9d8186ef0d20 1 lru 00000080:00200000:2.0:1775764923.497419:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000094f30dde 1: 1< 0 - - > 2< 4096 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497420:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008457470c 00000080:00200000:2.0:1775764923.497421:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 2. 00000080:00200000:2.0:1775764923.497421:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 2 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497429:0:34105:0:(rw26.c:858:ll_write_end()) pos 8192, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497432:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000436f09b7[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497433:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006de2fd28 17ffffc0001001 3:0 ffff9d8186ef07e0 2 lru 00000080:00200000:2.0:1775764923.497436:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000e23b068 2: 1< 0 - - > 2< 8192 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497437:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000436f09b7 00000080:00200000:2.0:1775764923.497438:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 3. 00000080:00200000:2.0:1775764923.497439:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 3 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497445:0:34105:0:(rw26.c:858:ll_write_end()) pos 12288, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497448:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001abedb3e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497449:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000857a2ccd 17ffffc0001001 3:0 ffff9d8186ef08c0 3 lru 00000080:00200000:2.0:1775764923.497452:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003092bbe4 3: 1< 0 - - > 2< 12288 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497454:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001abedb3e 00000080:00200000:2.0:1775764923.497454:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 4. 00000080:00200000:2.0:1775764923.497455:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 4 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497464:0:34105:0:(rw26.c:858:ll_write_end()) pos 16384, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497466:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000070bb8dc2[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497468:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000412eb10b 17ffffc0001001 3:0 ffff9d8186ef0c40 4 lru 00000080:00200000:2.0:1775764923.497471:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000032113681 4: 1< 0 - - > 2< 16384 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497472:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000070bb8dc2 00000080:00200000:2.0:1775764923.497473:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 5. 00000080:00200000:2.0:1775764923.497474:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 5 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497480:0:34105:0:(rw26.c:858:ll_write_end()) pos 20480, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497482:0:34105:0:(rw26.c:916:ll_write_end()) page@000000004bfb80e1[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497484:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002f960e79 17ffffc0001001 3:0 ffff9d8186ef0a80 5 lru 00000080:00200000:2.0:1775764923.497487:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000eb8f865 5: 1< 0 - - > 2< 20480 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497487:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000004bfb80e1 00000080:00200000:2.0:1775764923.497488:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 6. 00000080:00200000:2.0:1775764923.497489:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 6 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497495:0:34105:0:(rw26.c:858:ll_write_end()) pos 24576, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497497:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006d1139ed[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497499:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e2e640a8 17ffffc0001001 3:0 ffff9d8186ef0380 6 lru 00000080:00200000:2.0:1775764923.497502:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ef82da21 6: 1< 0 - - > 2< 24576 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497503:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006d1139ed 00000080:00200000:2.0:1775764923.497503:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 7. 00000080:00200000:2.0:1775764923.497504:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 7 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497510:0:34105:0:(rw26.c:858:ll_write_end()) pos 28672, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497512:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000076fb7598[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497514:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003e6ee01c 17ffffc0001001 3:0 ffff9d8186ef0e00 7 lru 00000080:00200000:2.0:1775764923.497517:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000043614c6f 7: 1< 0 - - > 2< 28672 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497518:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000076fb7598 00000080:00200000:2.0:1775764923.497518:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 8. 00000080:00200000:2.0:1775764923.497519:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 8 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497532:0:34105:0:(rw26.c:858:ll_write_end()) pos 32768, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497536:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003228c133[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497539:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c58fc72f 17ffffc0001001 3:0 ffff9d8186ef01c0 8 lru 00000080:00200000:2.0:1775764923.497544:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ddf2f6f5 8: 1< 0 - - > 2< 32768 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497546:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003228c133 00000080:00200000:2.0:1775764923.497547:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 9. 00000080:00200000:2.0:1775764923.497549:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 9 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497563:0:34105:0:(rw26.c:858:ll_write_end()) pos 36864, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497567:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a4033876[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497570:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bd8d8221 17ffffc0001001 3:0 ffff9d8186ef0700 9 lru 00000080:00200000:2.0:1775764923.497575:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000770ca3f4 9: 1< 0 - - > 2< 36864 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497577:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a4033876 00000080:00200000:2.0:1775764923.497578:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 10. 00000080:00200000:2.0:1775764923.497580:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 10 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497594:0:34105:0:(rw26.c:858:ll_write_end()) pos 40960, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497598:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000a1dbd937[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497601:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000243f2ccf 17ffffc0001001 3:0 ffff9d8186ef0620 10 lru 00000080:00200000:2.0:1775764923.497606:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000fdce73d6 10: 1< 0 - - > 2< 40960 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497608:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000a1dbd937 00000080:00200000:2.0:1775764923.497609:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 11. 00000080:00200000:2.0:1775764923.497611:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 11 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497641:0:34105:0:(rw26.c:858:ll_write_end()) pos 45056, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497645:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002f55844c[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497648:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d49a7ba5 17ffffc0001021 2:0 ffff9d8190440b60 11 lru 00000080:00200000:2.0:1775764923.497653:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000006b0ea94c 11: 1< 0 - - > 2< 45056 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497654:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002f55844c 00000080:00200000:2.0:1775764923.497655:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 12. 00000080:00200000:2.0:1775764923.497657:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 12 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497671:0:34105:0:(rw26.c:858:ll_write_end()) pos 49152, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497675:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c5b7b623[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497678:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009d1f808c 17ffffc0001001 3:0 ffff9d81904409a0 12 lru 00000080:00200000:2.0:1775764923.497683:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dc85cc83 12: 1< 0 - - > 2< 49152 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497685:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c5b7b623 00000080:00200000:2.0:1775764923.497686:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 13. 00000080:00200000:2.0:1775764923.497688:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 13 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497702:0:34105:0:(rw26.c:858:ll_write_end()) pos 53248, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497706:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b9f19b5e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497708:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001101b227 17ffffc0001001 3:0 ffff9d8190440540 13 lru 00000080:00200000:2.0:1775764923.497713:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003e7ce6eb 13: 1< 0 - - > 2< 53248 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497715:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b9f19b5e 00000080:00200000:2.0:1775764923.497716:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 14. 00000080:00200000:2.0:1775764923.497718:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 14 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497730:0:34105:0:(rw26.c:858:ll_write_end()) pos 57344, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497734:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000394a6bbd[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497737:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000082011868 17ffffc0001001 3:0 ffff9d8190440000 14 lru 00000080:00200000:2.0:1775764923.497742:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000075fa9947 14: 1< 0 - - > 2< 57344 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497743:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000394a6bbd 00000080:00200000:2.0:1775764923.497745:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 15. 00000080:00200000:2.0:1775764923.497746:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 15 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497760:0:34105:0:(rw26.c:858:ll_write_end()) pos 61440, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497764:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d41d4bae[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497768:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d598a823 17ffffc0001001 3:0 ffff9d81904402a0 15 lru 00000080:00200000:2.0:1775764923.497773:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003e4f9025 15: 1< 0 - - > 2< 61440 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497776:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d41d4bae 00000080:00200000:2.0:1775764923.497777:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 16. 00000080:00200000:2.0:1775764923.497779:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 16 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497794:0:34105:0:(rw26.c:858:ll_write_end()) pos 65536, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497798:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001d464c08[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497801:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000003932721b 17ffffc0001001 3:0 ffff9d8190440ee0 16 lru 00000080:00200000:2.0:1775764923.497807:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001047e585 16: 1< 0 - - > 2< 65536 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497809:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001d464c08 00000080:00200000:2.0:1775764923.497810:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 17. 00000080:00200000:2.0:1775764923.497812:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 17 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497828:0:34105:0:(rw26.c:858:ll_write_end()) pos 69632, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497833:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000604af613[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497835:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000eed4834e 17ffffc0001001 3:0 ffff9d8190440d20 17 lru 00000080:00200000:2.0:1775764923.497842:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000b15246f3 17: 1< 0 - - > 2< 69632 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000100:00100000:1.0:1775764923.497843:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190511f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066224128:192.168.202.105@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1775764923.497847:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000604af613 00000080:00200000:2.0:1775764923.497850:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 18. 00000080:00200000:2.0:1775764923.497853:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 18 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497870:0:34105:0:(rw26.c:858:ll_write_end()) pos 73728, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497894:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000058ef8a11[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497897:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000042243a7d 17ffffc0001001 3:0 ffff9d81904407e0 18 lru 00000080:00200000:2.0:1775764923.497902:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d3f68c6c 18: 1< 0 - - > 2< 73728 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497904:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000058ef8a11 00000080:00200000:2.0:1775764923.497905:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 19. 00000080:00200000:2.0:1775764923.497907:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 19 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497921:0:34105:0:(rw26.c:858:ll_write_end()) pos 77824, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497925:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003e2313e5[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497927:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b6625046 17ffffc0001001 3:0 ffff9d81904408c0 19 lru 00000080:00200000:2.0:1775764923.497930:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d8f21ff7 19: 1< 0 - - > 2< 77824 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497932:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003e2313e5 00000080:00200000:2.0:1775764923.497932:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 20. 00000080:00200000:2.0:1775764923.497934:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 20 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497941:0:34105:0:(rw26.c:858:ll_write_end()) pos 81920, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497943:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000099e2961e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497945:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000229374e6 17ffffc0001001 3:0 ffff9d8190440c40 20 lru 00000080:00200000:2.0:1775764923.497948:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008697d001 20: 1< 0 - - > 2< 81920 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497950:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000099e2961e 00000080:00200000:2.0:1775764923.497950:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 21. 00000080:00200000:2.0:1775764923.497951:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 21 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497961:0:34105:0:(rw26.c:858:ll_write_end()) pos 86016, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497963:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008e93305e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497964:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000125aa179 17ffffc0001001 3:0 ffff9d8190440a80 21 lru 00000080:00200000:2.0:1775764923.497967:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a5487b32 21: 1< 0 - - > 2< 86016 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497968:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008e93305e 00000080:00200000:2.0:1775764923.497969:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 22. 00000080:00200000:2.0:1775764923.497970:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 22 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.497982:0:34105:0:(rw26.c:858:ll_write_end()) pos 90112, len 4096, copied 4096 00000080:00200000:2.0:1775764923.497985:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f9b30ec4[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.497986:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f655f1ea 17ffffc0001001 3:0 ffff9d8190440380 22 lru 00000080:00200000:2.0:1775764923.497990:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a2b91ab6 22: 1< 0 - - > 2< 90112 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.497991:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f9b30ec4 00000080:00200000:2.0:1775764923.497992:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 23. 00000080:00200000:2.0:1775764923.497993:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 23 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498001:0:34105:0:(rw26.c:858:ll_write_end()) pos 94208, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498003:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ffa0b1da[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498004:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000065da6b4c 17ffffc0001001 3:0 ffff9d8190440e00 23 lru 00000080:00200000:2.0:1775764923.498007:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c6ad04f3 23: 1< 0 - - > 2< 94208 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498008:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ffa0b1da 00000080:00200000:2.0:1775764923.498009:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 24. 00000080:00200000:2.0:1775764923.498010:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 24 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498017:0:34105:0:(rw26.c:858:ll_write_end()) pos 98304, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498019:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000d04ddae2[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498020:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000003214ece 17ffffc0001001 3:0 ffff9d81904401c0 24 lru 00000080:00200000:2.0:1775764923.498023:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000071ce21f0 24: 1< 0 - - > 2< 98304 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498025:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000d04ddae2 00000080:00200000:2.0:1775764923.498026:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 25. 00000080:00200000:2.0:1775764923.498026:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 25 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498034:0:34105:0:(rw26.c:858:ll_write_end()) pos 102400, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498036:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000631cd41d[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498037:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000044e82728 17ffffc0001001 3:0 ffff9d8190440700 25 lru 00000080:00200000:2.0:1775764923.498040:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000022909a03 25: 1< 0 - - > 2< 102400 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498042:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000631cd41d 00000080:00200000:2.0:1775764923.498043:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 26. 00000080:00200000:2.0:1775764923.498044:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 26 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498052:0:34105:0:(rw26.c:858:ll_write_end()) pos 106496, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498055:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cc245ce0[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498056:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f6346b77 17ffffc0001021 2:0 ffff9d8190440620 26 lru 00000080:00200000:2.0:1775764923.498060:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003ed47006 26: 1< 0 - - > 2< 106496 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498061:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cc245ce0 00000080:00200000:2.0:1775764923.498062:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 27. 00000080:00200000:2.0:1775764923.498063:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 27 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498070:0:34105:0:(rw26.c:858:ll_write_end()) pos 110592, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498072:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c9a98c29[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498073:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000aed34d6c 17ffffc0001001 3:0 ffff9d8190440460 27 lru 00000080:00200000:2.0:1775764923.498076:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007ee4df26 27: 1< 0 - - > 2< 110592 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498077:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c9a98c29 00000080:00200000:2.0:1775764923.498078:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 28. 00000080:00200000:2.0:1775764923.498079:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 28 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498085:0:34105:0:(rw26.c:858:ll_write_end()) pos 114688, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498087:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000091cfa0d8[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498089:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000065a73d1e 17ffffc0001001 3:0 ffff9d81904400e0 28 lru 00000080:00200000:2.0:1775764923.498092:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000286a9840 28: 1< 0 - - > 2< 114688 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498093:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000091cfa0d8 00000080:00200000:2.0:1775764923.498093:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 29. 00000080:00200000:2.0:1775764923.498094:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 29 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498107:0:34105:0:(rw26.c:858:ll_write_end()) pos 118784, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498109:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006faf64d8[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498111:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000004bfbefa2 17ffffc0001001 3:0 ffff9d81877251c0 29 lru 00000080:00200000:2.0:1775764923.498116:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000572fb172 29: 1< 0 - - > 2< 118784 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498118:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006faf64d8 00000080:00200000:2.0:1775764923.498119:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 30. 00000080:00200000:2.0:1775764923.498121:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 30 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498132:0:34105:0:(rw26.c:858:ll_write_end()) pos 122880, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498136:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c0d9c943[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498138:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000628cafcc 17ffffc0001001 3:0 ffff9d8187725700 30 lru 00000080:00200000:2.0:1775764923.498143:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000007753509b 30: 1< 0 - - > 2< 122880 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498145:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c0d9c943 00000080:00200000:2.0:1775764923.498146:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 31. 00000080:00200000:2.0:1775764923.498148:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 31 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498160:0:34105:0:(rw26.c:858:ll_write_end()) pos 126976, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498164:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007ad7318e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498167:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a4c1d75e 17ffffc0001001 3:0 ffff9d8187725620 31 lru 00000080:00200000:2.0:1775764923.498185:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003631a1b6 31: 1< 0 - - > 2< 126976 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498187:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007ad7318e 00000080:00200000:2.0:1775764923.498188:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 32. 00000080:00200000:2.0:1775764923.498190:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 32 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498202:0:34105:0:(rw26.c:858:ll_write_end()) pos 131072, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498204:0:34105:0:(rw26.c:916:ll_write_end()) page@000000007cb565b8[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498206:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000001bbccb47 17ffffc0001001 3:0 ffff9d8187725460 32 lru 00000080:00200000:2.0:1775764923.498209:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d77495cb 32: 1< 0 - - > 2< 131072 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498210:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000007cb565b8 00000080:00200000:2.0:1775764923.498211:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 33. 00000080:00200000:2.0:1775764923.498212:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 33 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498219:0:34105:0:(rw26.c:858:ll_write_end()) pos 135168, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498221:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ab9b5923[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498223:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000010236f91 17ffffc0001001 3:0 ffff9d81877250e0 33 lru 00000080:00200000:2.0:1775764923.498226:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000093592de 33: 1< 0 - - > 2< 135168 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498228:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ab9b5923 00000080:00200000:2.0:1775764923.498228:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 34. 00000080:00200000:2.0:1775764923.498229:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 34 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498236:0:34105:0:(rw26.c:858:ll_write_end()) pos 139264, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498238:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f450e8ee[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498239:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000007d7f145 17ffffc0001001 3:0 ffff9d8187725b60 34 lru 00000080:00200000:2.0:1775764923.498242:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001fc3a762 34: 1< 0 - - > 2< 139264 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498243:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f450e8ee 00000080:00200000:2.0:1775764923.498244:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 35. 00000080:00200000:2.0:1775764923.498244:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 35 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498251:0:34105:0:(rw26.c:858:ll_write_end()) pos 143360, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498253:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008fdbb421[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498254:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e7e6103e 17ffffc0001001 3:0 ffff9d81877259a0 35 lru 00000080:00200000:2.0:1775764923.498257:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000064506ebe 35: 1< 0 - - > 2< 143360 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498259:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008fdbb421 00000080:00200000:2.0:1775764923.498260:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 36. 00000080:00200000:2.0:1775764923.498261:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 36 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498267:0:34105:0:(rw26.c:858:ll_write_end()) pos 147456, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498269:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000063f280fd[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498271:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000009ac8635 17ffffc0001001 3:0 ffff9d8187725540 36 lru 00000080:00200000:2.0:1775764923.498274:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a556c3eb 36: 1< 0 - - > 2< 147456 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498275:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000063f280fd 00000080:00200000:2.0:1775764923.498275:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 37. 00000080:00200000:2.0:1775764923.498276:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 37 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498282:0:34105:0:(rw26.c:858:ll_write_end()) pos 151552, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498284:0:34105:0:(rw26.c:916:ll_write_end()) page@000000006ea6d61d[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498286:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000006de39e1d 17ffffc0001001 3:0 ffff9d8187725000 37 lru 00000080:00200000:2.0:1775764923.498289:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000eb41a438 37: 1< 0 - - > 2< 151552 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498290:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000006ea6d61d 00000080:00200000:2.0:1775764923.498291:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 38. 00000080:00200000:2.0:1775764923.498292:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 38 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498298:0:34105:0:(rw26.c:858:ll_write_end()) pos 155648, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498300:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000008069a56[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498301:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000791d7309 17ffffc0001001 3:0 ffff9d81877252a0 38 lru 00000080:00200000:2.0:1775764923.498304:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000157b2b5e 38: 1< 0 - - > 2< 155648 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498305:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000008069a56 00000080:00200000:2.0:1775764923.498306:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 39. 00000080:00200000:2.0:1775764923.498307:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 39 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498314:0:34105:0:(rw26.c:858:ll_write_end()) pos 159744, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498316:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000073158dca[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498319:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000032c58f30 17ffffc0001001 3:0 ffff9d8187725ee0 39 lru 00000080:00200000:2.0:1775764923.498324:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000004188add4 39: 1< 0 - - > 2< 159744 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498325:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000073158dca 00000080:00200000:2.0:1775764923.498327:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 40. 00000080:00200000:2.0:1775764923.498328:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 40 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498341:0:34105:0:(rw26.c:858:ll_write_end()) pos 163840, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498345:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f6141b4e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498347:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000f7cbea88 17ffffc0001001 3:0 ffff9d8187725d20 40 lru 00000080:00200000:2.0:1775764923.498352:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d3ec6f4c 40: 1< 0 - - > 2< 163840 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498355:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f6141b4e 00000080:00200000:2.0:1775764923.498356:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 41. 00000080:00200000:2.0:1775764923.498357:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 41 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498373:0:34105:0:(rw26.c:858:ll_write_end()) pos 167936, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498376:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000969e308d[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498380:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a5f81f6e 17ffffc0001021 2:0 ffff9d81877257e0 41 lru 00000080:00200000:2.0:1775764923.498383:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f9e4bf93 41: 1< 0 - - > 2< 167936 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498385:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000969e308d 00000080:00200000:2.0:1775764923.498385:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 42. 00000080:00200000:2.0:1775764923.498387:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 42 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498396:0:34105:0:(rw26.c:858:ll_write_end()) pos 172032, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498398:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001028ed66[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498399:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000e06335e2 17ffffc0001001 3:0 ffff9d81877258c0 42 lru 00000080:00200000:2.0:1775764923.498403:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000259dd643 42: 1< 0 - - > 2< 172032 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498404:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001028ed66 00000080:00200000:2.0:1775764923.498405:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 43. 00000080:00200000:2.0:1775764923.498406:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 43 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498412:0:34105:0:(rw26.c:858:ll_write_end()) pos 176128, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498414:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002dd6d52e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498416:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000ca6f3d0a 17ffffc0001001 3:0 ffff9d8187725c40 43 lru 00000080:00200000:2.0:1775764923.498419:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cb49c23a 43: 1< 0 - - > 2< 176128 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498420:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002dd6d52e 00000080:00200000:2.0:1775764923.498420:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 44. 00000080:00200000:2.0:1775764923.498421:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 44 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498428:0:34105:0:(rw26.c:858:ll_write_end()) pos 180224, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498430:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f529cda3[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498431:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000050bb85e4 17ffffc0001001 3:0 ffff9d8187725a80 44 lru 00000080:00200000:2.0:1775764923.498434:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a1b2e460 44: 1< 0 - - > 2< 180224 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498436:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f529cda3 00000080:00200000:2.0:1775764923.498437:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 45. 00000080:00200000:2.0:1775764923.498438:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 45 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498444:0:34105:0:(rw26.c:858:ll_write_end()) pos 184320, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498446:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000216e25e4[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498448:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009a2fb26f 17ffffc0001001 3:0 ffff9d8187725380 45 lru 00000080:00200000:2.0:1775764923.498451:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000dacd9f7c 45: 1< 0 - - > 2< 184320 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498452:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000216e25e4 00000080:00200000:2.0:1775764923.498453:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 46. 00000080:00200000:2.0:1775764923.498453:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 46 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498461:0:34105:0:(rw26.c:858:ll_write_end()) pos 188416, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498463:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000047d3516f[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498465:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005eb53a28 17ffffc0001001 3:0 ffff9d8187725e00 46 lru 00000080:00200000:2.0:1775764923.498468:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c0a45f32 46: 1< 0 - - > 2< 188416 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498469:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000047d3516f 00000080:00200000:2.0:1775764923.498470:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 47. 00000080:00200000:2.0:1775764923.498471:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 47 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498482:0:34105:0:(rw26.c:858:ll_write_end()) pos 192512, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498484:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000059e0877d[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498486:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000024b9804e 17ffffc0001001 3:0 ffff9d81846d2540 47 lru 00000080:00200000:2.0:1775764923.498489:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f0ae1312 47: 1< 0 - - > 2< 192512 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498490:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000059e0877d 00000080:00200000:2.0:1775764923.498491:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 48. 00000080:00200000:2.0:1775764923.498492:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 48 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498499:0:34105:0:(rw26.c:858:ll_write_end()) pos 196608, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498501:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008577cc6d[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498502:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000500bb532 17ffffc0001001 3:0 ffff9d81846d2000 48 lru 00000080:00200000:2.0:1775764923.498506:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000008668941f 48: 1< 0 - - > 2< 196608 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498507:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008577cc6d 00000080:00200000:2.0:1775764923.498507:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 49. 00000080:00200000:2.0:1775764923.498508:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 49 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498516:0:34105:0:(rw26.c:858:ll_write_end()) pos 200704, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498519:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b1fbcd39[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498521:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000c4947c6d 17ffffc0001001 3:0 ffff9d81846d22a0 49 lru 00000080:00200000:2.0:1775764923.498526:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000000330d379 49: 1< 0 - - > 2< 200704 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498528:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b1fbcd39 00000080:00200000:2.0:1775764923.498530:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 50. 00000080:00200000:2.0:1775764923.498531:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 50 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498545:0:34105:0:(rw26.c:858:ll_write_end()) pos 204800, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498549:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b605498a[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498552:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000bd8fa646 17ffffc0001001 3:0 ffff9d81846d2ee0 50 lru 00000080:00200000:2.0:1775764923.498557:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ec46e33a 50: 1< 0 - - > 2< 204800 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498559:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b605498a 00000080:00200000:2.0:1775764923.498560:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 51. 00000080:00200000:2.0:1775764923.498562:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 51 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498575:0:34105:0:(rw26.c:858:ll_write_end()) pos 208896, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498579:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000e34cadb[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498582:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d3bf96de 17ffffc0001001 3:0 ffff9d81846d2d20 51 lru 00000080:00200000:2.0:1775764923.498587:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000039bc8a13 51: 1< 0 - - > 2< 208896 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498589:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000e34cadb 00000080:00200000:2.0:1775764923.498590:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 52. 00000080:00200000:2.0:1775764923.498591:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 52 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498607:0:34105:0:(rw26.c:858:ll_write_end()) pos 212992, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498612:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000cb5d21fc[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498614:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000079a7dfa8 17ffffc0001001 3:0 ffff9d81846d27e0 52 lru 00000080:00200000:2.0:1775764923.498619:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000cb8ac6ce 52: 1< 0 - - > 2< 212992 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498621:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000cb5d21fc 00000080:00200000:2.0:1775764923.498622:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 53. 00000080:00200000:2.0:1775764923.498624:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 53 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498637:0:34105:0:(rw26.c:858:ll_write_end()) pos 217088, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498641:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c6023a45[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498644:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000010af38fd 17ffffc0001001 3:0 ffff9d81846d28c0 53 lru 00000080:00200000:2.0:1775764923.498649:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c4a1c388 53: 1< 0 - - > 2< 217088 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498651:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c6023a45 00000080:00200000:2.0:1775764923.498652:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 54. 00000080:00200000:2.0:1775764923.498654:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 54 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498668:0:34105:0:(rw26.c:858:ll_write_end()) pos 221184, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498672:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b2b4132d[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498675:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000098dc569d 17ffffc0001001 3:0 ffff9d81846d2c40 54 lru 00000080:00200000:2.0:1775764923.498680:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000097f17427 54: 1< 0 - - > 2< 221184 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498682:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b2b4132d 00000080:00200000:2.0:1775764923.498683:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 55. 00000080:00200000:2.0:1775764923.498684:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 55 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498698:0:34105:0:(rw26.c:858:ll_write_end()) pos 225280, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498701:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005fca9e51[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498704:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a6f68fa3 17ffffc0001001 3:0 ffff9d81846d2a80 55 lru 00000080:00200000:2.0:1775764923.498709:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bbe1c1db 55: 1< 0 - - > 2< 225280 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498711:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005fca9e51 00000080:00200000:2.0:1775764923.498712:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 56. 00000080:00200000:2.0:1775764923.498714:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 56 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498730:0:34105:0:(rw26.c:858:ll_write_end()) pos 229376, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498734:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009a1f0265[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498737:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d9ab177d 17ffffc0001021 2:0 ffff9d81846d2380 56 lru 00000080:00200000:2.0:1775764923.498742:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000060f6738d 56: 1< 0 - - > 2< 229376 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498744:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009a1f0265 00000080:00200000:2.0:1775764923.498745:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 57. 00000080:00200000:2.0:1775764923.498747:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 57 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498761:0:34105:0:(rw26.c:858:ll_write_end()) pos 233472, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498765:0:34105:0:(rw26.c:916:ll_write_end()) page@000000003d1fe218[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498767:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000aef8490a 17ffffc0001001 3:0 ffff9d81846d2e00 57 lru 00000080:00200000:2.0:1775764923.498772:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ff877fda 57: 1< 0 - - > 2< 233472 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498774:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000003d1fe218 00000080:00200000:2.0:1775764923.498775:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 58. 00000080:00200000:2.0:1775764923.498777:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 58 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498790:0:34105:0:(rw26.c:858:ll_write_end()) pos 237568, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498793:0:34105:0:(rw26.c:916:ll_write_end()) page@000000001811892a[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498795:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000077fee909 17ffffc0001001 3:0 ffff9d81846d21c0 58 lru 00000080:00200000:2.0:1775764923.498798:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000034477655 58: 1< 0 - - > 2< 237568 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498800:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000001811892a 00000080:00200000:2.0:1775764923.498800:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 59. 00000080:00200000:2.0:1775764923.498801:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 59 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498809:0:34105:0:(rw26.c:858:ll_write_end()) pos 241664, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498811:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000726fb01a[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498812:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000052b5abb3 17ffffc0001001 3:0 ffff9d81846d2700 59 lru 00000080:00200000:2.0:1775764923.498815:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000171b125d 59: 1< 0 - - > 2< 241664 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498816:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000726fb01a 00000080:00200000:2.0:1775764923.498817:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 60. 00000080:00200000:2.0:1775764923.498818:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 60 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498826:0:34105:0:(rw26.c:858:ll_write_end()) pos 245760, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498828:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000ada0b4dc[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498831:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000a81bbc80 17ffffc0001001 3:0 ffff9d81846d2620 60 lru 00000080:00200000:2.0:1775764923.498835:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001366e212 60: 1< 0 - - > 2< 245760 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498838:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000ada0b4dc 00000080:00200000:2.0:1775764923.498839:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 61. 00000080:00200000:2.0:1775764923.498840:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 61 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498852:0:34105:0:(rw26.c:858:ll_write_end()) pos 249856, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498858:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000010e9572b[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498860:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000d6f7afaf 17ffffc0001001 3:0 ffff9d81846d2460 61 lru 00000080:00200000:2.0:1775764923.498864:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000d58d32d2 61: 1< 0 - - > 2< 249856 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498866:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000010e9572b 00000080:00200000:2.0:1775764923.498867:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 62. 00000080:00200000:2.0:1775764923.498869:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 62 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498901:0:34105:0:(rw26.c:858:ll_write_end()) pos 253952, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498905:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000090f0fcad[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498908:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000002e022424 17ffffc0001001 3:0 ffff9d81846d20e0 62 lru 00000080:00200000:2.0:1775764923.498913:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000001f3ee7b1 62: 1< 0 - - > 2< 253952 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498916:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000090f0fcad 00000080:00200000:2.0:1775764923.498917:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 63. 00000080:00200000:2.0:1775764923.498920:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 63 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498934:0:34105:0:(rw26.c:858:ll_write_end()) pos 258048, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498937:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000f6e0adf3[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498940:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000092e7eab 17ffffc0001001 3:0 ffff9d81846d2b60 63 lru 00000080:00200000:2.0:1775764923.498944:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a331ed3f 63: 1< 0 - - > 2< 258048 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498945:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000f6e0adf3 00000080:00200000:2.0:1775764923.498946:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 64. 00000080:00200000:2.0:1775764923.498947:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 64 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498967:0:34105:0:(rw26.c:858:ll_write_end()) pos 262144, len 4096, copied 4096 00000080:00200000:2.0:1775764923.498970:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000271412a6[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.498972:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000071fcf7fc 17ffffc0001001 3:0 ffff9d81846d29a0 64 lru 00000080:00200000:2.0:1775764923.498976:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000256641dc 64: 1< 0 - - > 2< 262144 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.498978:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000271412a6 00000080:00200000:2.0:1775764923.498979:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 65. 00000080:00200000:2.0:1775764923.498980:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 65 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.498998:0:34105:0:(rw26.c:858:ll_write_end()) pos 266240, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499001:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000073ac8ec0[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499003:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000008780fbee 17ffffc0001001 3:0 ffff9d81902132a0 65 lru 00000080:00200000:2.0:1775764923.499008:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000010634c4c 65: 1< 0 - - > 2< 266240 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499010:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000073ac8ec0 00000080:00200000:2.0:1775764923.499011:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 66. 00000080:00200000:2.0:1775764923.499012:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 66 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499024:0:34105:0:(rw26.c:858:ll_write_end()) pos 270336, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499027:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000af275da6[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499030:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000985c9dde 17ffffc0001001 3:0 ffff9d8190213ee0 66 lru 00000080:00200000:2.0:1775764923.499034:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000730b9d14 66: 1< 0 - - > 2< 270336 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499036:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000af275da6 00000080:00200000:2.0:1775764923.499037:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 67. 00000080:00200000:2.0:1775764923.499038:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 67 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499051:0:34105:0:(rw26.c:858:ll_write_end()) pos 274432, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499055:0:34105:0:(rw26.c:916:ll_write_end()) page@000000002bbc299e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499057:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000546d7dc4 17ffffc0001001 3:0 ffff9d8190213d20 67 lru 00000080:00200000:2.0:1775764923.499062:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000a6905d36 67: 1< 0 - - > 2< 274432 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499064:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000002bbc299e 00000080:00200000:2.0:1775764923.499064:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 68. 00000080:00200000:2.0:1775764923.499066:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 68 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499075:0:34105:0:(rw26.c:858:ll_write_end()) pos 278528, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499077:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000025f4a100[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499078:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009d3f5c61 17ffffc0001001 3:0 ffff9d81902137e0 68 lru 00000080:00200000:2.0:1775764923.499082:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000003ba91b22 68: 1< 0 - - > 2< 278528 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499083:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000025f4a100 00000080:00200000:2.0:1775764923.499083:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 69. 00000080:00200000:2.0:1775764923.499084:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 69 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499091:0:34105:0:(rw26.c:858:ll_write_end()) pos 282624, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499093:0:34105:0:(rw26.c:916:ll_write_end()) page@000000008b56780e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499094:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000007dccf08f 17ffffc0001001 3:0 ffff9d81902138c0 69 lru 00000080:00200000:2.0:1775764923.499097:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000bbffd6dd 69: 1< 0 - - > 2< 282624 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499098:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000008b56780e 00000080:00200000:2.0:1775764923.499099:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 70. 00000080:00200000:2.0:1775764923.499100:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 70 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499107:0:34105:0:(rw26.c:858:ll_write_end()) pos 286720, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499109:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000b866b415[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499110:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005df578b4 17ffffc0001001 3:0 ffff9d8190213c40 70 lru 00000080:00200000:2.0:1775764923.499113:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000083ab4f55 70: 1< 0 - - > 2< 286720 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499115:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000b866b415 00000080:00200000:2.0:1775764923.499115:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 71. 00000080:00200000:2.0:1775764923.499116:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 71 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499125:0:34105:0:(rw26.c:858:ll_write_end()) pos 290816, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499127:0:34105:0:(rw26.c:916:ll_write_end()) page@00000000c2577f17[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499128:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000020d29a43 17ffffc0001021 2:0 ffff9d8190213a80 71 lru 00000080:00200000:2.0:1775764923.499131:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@000000009c3c93c1 71: 1< 0 - - > 2< 290816 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499133:0:34105:0:(rw26.c:916:ll_write_end()) end page@00000000c2577f17 00000080:00200000:2.0:1775764923.499134:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 72. 00000080:00200000:2.0:1775764923.499134:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 72 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499141:0:34105:0:(rw26.c:858:ll_write_end()) pos 294912, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499143:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000077231f0e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499145:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000005d4420d4 17ffffc0001001 3:0 ffff9d8190213380 72 lru 00000080:00200000:2.0:1775764923.499148:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000c5132331 72: 1< 0 - - > 2< 294912 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499149:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000077231f0e 00000080:00200000:2.0:1775764923.499150:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 73. 00000080:00200000:2.0:1775764923.499151:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 73 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499157:0:34105:0:(rw26.c:858:ll_write_end()) pos 299008, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499159:0:34105:0:(rw26.c:916:ll_write_end()) page@000000009985db95[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499161:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000018c3c31c 17ffffc0001001 3:0 ffff9d8190213e00 73 lru 00000080:00200000:2.0:1775764923.499164:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000160aa117 73: 1< 0 - - > 2< 299008 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499165:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000009985db95 00000080:00200000:2.0:1775764923.499166:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 74. 00000080:00200000:2.0:1775764923.499167:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 74 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499189:0:34105:0:(rw26.c:858:ll_write_end()) pos 303104, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499192:0:34105:0:(rw26.c:916:ll_write_end()) page@000000005b2ddbb5[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499193:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @0000000017f0410a 17ffffc0001001 3:0 ffff9d81902131c0 74 lru 00000080:00200000:2.0:1775764923.499196:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@0000000056f71591 74: 1< 0 - - > 2< 303104 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499197:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000005b2ddbb5 00000080:00200000:2.0:1775764923.499198:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 75. 00000080:00200000:2.0:1775764923.499199:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 75 of 0 to 4096 bytes 00000080:00200000:2.0:1775764923.499206:0:34105:0:(rw26.c:858:ll_write_end()) pos 307200, len 4096, copied 4096 00000080:00200000:2.0:1775764923.499208:0:34105:0:(rw26.c:916:ll_write_end()) page@000000000122b51e[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499209:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @00000000b1d630ea 17ffffc0001001 3:0 ffff9d8190213700 75 lru 00000080:00200000:2.0:1775764923.499212:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000f0406224 75: 1< 0 - - > 2< 307200 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499213:0:34105:0:(rw26.c:916:ll_write_end()) end page@000000000122b51e 00000080:00200000:2.0:1775764923.499214:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 76. 00000080:00200000:2.0:1775764923.499215:0:34105:0:(rw26.c:663:ll_write_begin()) Writing 76 of 0 to 1767 bytes 00000080:00200000:2.0:1775764923.499226:0:34105:0:(rw26.c:858:ll_write_end()) pos 311296, len 1767, copied 1767 00000080:00200000:2.0:1775764923.499229:0:34105:0:(rw26.c:916:ll_write_end()) page@0000000052cf018b[3 00000000704f5606 2 1 00000000fae803bc] 00000080:00200000:2.0:1775764923.499230:0:34105:0:(rw26.c:916:ll_write_end()) vmpage @000000009fb7b0da 17ffffc0001001 3:0 ffff9d8190213620 76 lru 00000080:00200000:2.0:1775764923.499233:0:34105:0:(rw26.c:916:ll_write_end()) osc-page@00000000ff494c38 76: 1< 0 - - > 2< 311296 0 0 0x0 0x0 | 000000009452db5f 00000000064572f7 > 3< 0 0 > 4< 0 0 8 167936 - | - - + - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:2.0:1775764923.499235:0:34105:0:(rw26.c:916:ll_write_end()) end page@0000000052cf018b 00000080:00200000:2.0:1775764923.499236:0:34105:0:(rw26.c:916:ll_write_end()) queued cl_page: 77. 00000080:00200000:2.0:1775764923.499238:0:34105:0:(vvp_io.c:1167:vvp_io_write_commit()) commit async pages: 77, from 0, to 1767 prio 0 00000008:00010000:2.0:1775764923.499253:0:34105:0:(osc_cache.c:848:osc_extent_find()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499266:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499271:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499276:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499280:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499284:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499288:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499296:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499300:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499304:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499308:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499312:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499318:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499325:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499332:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.499345:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fc478, count 15, dirtied 15 00000008:00010000:2.0:1775764923.499351:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499360:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499368:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499377:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499382:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499387:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499391:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499395:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499399:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499406:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499410:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499414:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499418:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.499422:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.499432:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fc478, count 14, dirtied 14 00000008:00010000:2.0:1775764923.499493:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 000000007a5950c3 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775764923.499500:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 000000007a5950c3 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.499527:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00100000:2.0:1775764923.499578:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190512680 - object 0x280000401:546 offset 0<>903 00000008:00010000:2.0:1775764923.499597:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 0000000048d98cad ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000f2715abe/0xeef44110eff42ce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->385023) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152591 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775764923.499603:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 0000000048d98cad ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000f2715abe/0xeef44110eff42ce lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->385023) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152591 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1775764923.499626:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.499638:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190512680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066224256:192.168.202.105@tcp:4:cp.0 00010000:00010000:2.0:1775764923.499693:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000f2715abe/0xeef44110eff42ce lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x223:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->385023) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152591 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1775764923.499846:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190511180 - object 0x280000401:547 offset 0<>381426 00000008:00010000:2.0:1775764923.499864:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 00000000d56aa886 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fc930d1/0xeef44110eff4314 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152647 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775764923.499870:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 00000000d56aa886 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fc930d1/0xeef44110eff4314 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152647 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1775764923.499874:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190511180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066224384:192.168.202.105@tcp:4:cp.0 00010000:00010000:2.0:1775764923.499931:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000005fc930d1/0xeef44110eff4314 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x224:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152647 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1775764923.499963:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190511500 - object 0x280000401:548 offset 0<>9974 00000008:00010000:2.0:1775764923.499985:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 00000000c14d04a0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000f2acc1af/0xeef44110eff435a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321526fd expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775764923.499992:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 00000000c14d04a0 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000f2acc1af/0xeef44110eff435a lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321526fd expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1775764923.500002:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190511500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066224512:192.168.202.105@tcp:4:cp.0 00010000:00010000:2.0:1775764923.500027:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000f2acc1af/0xeef44110eff435a lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x225:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->40959) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321526fd expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1775764923.500085:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190510a80 - object 0x280000401:549 offset 0<>39343 00000008:00010000:2.0:1775764923.500108:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 0000000080002685 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000078c87789/0xeef44110eff43a0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321527b3 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775764923.500117:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 0000000080002685 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000078c87789/0xeef44110eff43a0 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321527b3 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1775764923.500135:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066224640:192.168.202.105@tcp:4:cp.0 00010000:00010000:2.0:1775764923.500138:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 0000000078c87789/0xeef44110eff43a0 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x226:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321527b3 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1775764923.500164:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190513800 - object 0x280000401:550 offset 0<>674 00000100:00100000:1.0:1775764923.500190:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:1.0:1775764923.500200:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190513800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066224768:192.168.202.105@tcp:4:cp.0 00000008:00010000:2.0:1775764923.500202:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 000000006b5713b7 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000001653533e/0xeef44110eff43e6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152869 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775764923.500214:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 000000006b5713b7 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000001653533e/0xeef44110eff43e6 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152869 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.500238:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000001653533e/0xeef44110eff43e6 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152869 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1775764923.500267:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190511880 - object 0x280000401:551 offset 0<>2663 00000008:00010000:2.0:1775764923.500286:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 000000007256560a ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e112710a/0xeef44110eff442c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215291f expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775764923.500296:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 000000007256560a ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e112710a/0xeef44110eff442c lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215291f expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.500330:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000e112710a/0xeef44110eff442c lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->28671) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa3215291f expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1775764923.500367:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190513100 - object 0x280000401:552 offset 0<>28245 00000008:00010000:2.0:1775764923.500383:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 00000000f3d5a5f6 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000919293c4/0xeef44110eff4472 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321529d5 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1775764923.500391:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190513100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066225024:192.168.202.105@tcp:4:cp.0 00000008:00010000:2.0:1775764923.500395:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 00000000f3d5a5f6 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000919293c4/0xeef44110eff4472 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321529d5 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.500422:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000919293c4/0xeef44110eff4472 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x229:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321529d5 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1775764923.500456:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190510e00 - object 0x280000401:553 offset 0<>1754 00000100:00100000:1.0:1775764923.500489:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066225152:192.168.202.105@tcp:4:cp.0 00000008:00010000:2.0:1775764923.500490:0:31861:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 0000000060f1ddbb ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ae543bea/0xeef44110eff44b8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x22a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152a8b expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775764923.500502:0:31861:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 0000000060f1ddbb ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ae543bea/0xeef44110eff44b8 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x22a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152a8b expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775764923.500526:0:31861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 00000000ae543bea/0xeef44110eff44b8 lrc: 3/0,0 mode: PW/PW res: [0x280000401:0x22a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152a8b expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1775764923.500562:0:31861:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190513480 - object 0x280000401:554 offset 0<>1571 00000008:00010000:1.0:1775764923.501425:0:10946:0:(osc_cache.c:901:osc_extent_finish()) ### extent: 000000007a5950c3 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00010000:1.0:1775764923.501473:0:10946:0:(osc_cache.c:496:__osc_extent_remove()) ### extent: 000000007a5950c3 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000009e04f1ab/0xeef44110eff4281 lrc: 2/0,0 mode: PW/PW res: [0x280000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa321524bf expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000100:00100000:1.0:1775764923.501500:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190512680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066224256:192.168.202.105@tcp:4:cp.0 00000008:00010000:2.0:1775764923.501501:0:34105:0:(osc_cache.c:564:osc_extent_merge()) ### extent: 000000006e0b2bb6 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501511:0:34105:0:(osc_cache.c:496:__osc_extent_remove()) ### extent: 000000006e0b2bb6 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501517:0:34105:0:(osc_cache.c:581:osc_extent_merge()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501522:0:34105:0:(osc_cache.c:836:osc_extent_find()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501533:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501540:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501545:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501550:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501560:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501565:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501570:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501575:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501580:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501584:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501592:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501602:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501611:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501621:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.501638:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fc478, count 15, dirtied 15 00000008:00010000:2.0:1775764923.501646:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501657:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501668:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501679:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501690:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501698:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501704:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501719:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501736:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501753:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501764:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501785:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501797:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501806:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501820:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.501834:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fc478, count 15, dirtied 15 00000008:00010000:2.0:1775764923.501840:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501849:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501859:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501894:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501906:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501914:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501922:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501929:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501934:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501942:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501953:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501959:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501966:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501976:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.501984:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.501997:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fc478, count 15, dirtied 15 00000008:00010000:2.0:1775764923.502003:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.502014:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1775764923.502028:0:34105:0:(osc_cache.c:2557:osc_queue_async_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1775764923.502037:0:34105:0:(vvp_io.c:1080:vvp_set_batch_dirty()) mapping ffff9d81843fc478, count 3, dirtied 3 00000080:00200000:2.0:1775764923.502041:0:34105:0:(vvp_io.c:1190:vvp_io_write_commit()) Committed 77 pages 313063 bytes, tot: 313063 00000080:00200000:2.0:1775764923.502044:0:34105:0:(vvp_io.c:1353:vvp_io_write_start()) test-framework.sh: write bytes 0, result: 313063 00000020:00010000:2.0:1775764923.502055:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_release():245 00000020:00010000:2.0:1775764923.502058:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1775764923.502061:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9d81876ce130 (ffff9d8191255758/1) at cl_lock_cancel():142 00010000:00010000:2.0:1775764923.502066:0:34105:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 5/0,1 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x810020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1775764923.502075:0:34105:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1775764923.502088:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d8191252e48 (ffff9d8191255518/0) at cl_lock_fini():72 00000020:00010000:2.0:1775764923.502091:0:34105:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9d81876ce130 (ffff9d8191255518/1) at cl_lock_fini():72 00000080:00200000:2.0:1775764923.502098:0:34105:0:(vvp_io.c:517:vvp_io_advance()) advancing 313063 bytes 00000080:00200000:2.0:1775764923.502104:0:34105:0:(file.c:2173:ll_file_io_generic()) Range unlock [0, 76] 00000080:00200000:2.0:1775764923.502121:0:34105:0:(vvp_io.c:288:vvp_io_fini()) [0x240004281:0x4e:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1775764923.502125:0:34105:0:(file.c:2206:ll_file_io_generic()) test-framework.sh: 2 io complete with rc: 0, result: 313063, restart: 0 00000080:00200000:2.0:1775764923.502133:0:34105:0:(file.c:1742:ll_merge_attr_nolock()) [0x240004281:0x4e:0x0] updating i_size 313063 i_blocks 0 00000080:00200000:2.0:1775764923.502148:0:34105:0:(file.c:2292:ll_file_io_generic()) iot: 2, result: 313063 00000080:00200008:2.0:1775764923.502201:0:34105:0:(file.c:447:ll_file_release()) START file test-framework.sh:[0x240004281:0x4e:0x0](ffff9d81843fc2c8), flags 100001 00000002:00100000:2.0:1775764923.502239:0:34105:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9d8190511c00 x1862023066223744/t38654705913(38654705913) o101->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 592/664 e 0 to 0 dl 1775764939 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1775764923.502257:0:34105:0:(genops.c:2442:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff9d81892c2000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1775764923.502262:0:34105:0:(client.c:806:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9d8190511f80 x1862023066225536/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1775764923.502280:0:34105:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190511f80 pname:cluuid:pid:xid:nid:opc:job cp:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:34105:1862023066225536:192.168.202.105@tcp:35:cp.0 00000100:00100000:2.0:1775764923.502328:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 11 seconds 00000100:00100000:1.0:1775764923.573083:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190708700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066225792:192.168.202.105@tcp:400:kworker.0 00010000:00010000:2.0:1775764924.212981:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764925.237042:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1775764925.621113:0:23872:0:(vvp_io.c:1817:vvp_io_init()) [0x200004a52:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1775764925.621125:0:23872:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1775764925.621193:0:23872:0:(vvp_io.c:288:vvp_io_fini()) [0x200004a52:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1775764925.621210:0:23872:0:(vvp_io.c:1817:vvp_io_init()) [0x200004a52:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1775764925.621215:0:23872:0:(lov_io.c:860:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:3.0:1775764925.621232:0:23872:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 000000009003ca74 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:3.0:1775764925.621246:0:23872:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 000000009003ca74 ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 2/0,0 mode: PW/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1775764925.621298:0:23872:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000005ddfc480/0xeef44110eff42ab lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x800020000020000 nid: local remote: 0x4dcde3aa32152536 expref: -99 pid: 34089 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000008:00100000:3.0:1775764925.621383:0:23872:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d819070a300 - object 0x2c0000401:546 offset 0<>5442 00000100:00100000:1.0:1775764925.621447:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d819070bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066226048:192.168.202.105@tcp:16:kworker.0 00010000:00010000:2.0:1775764926.260997:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764927.284995:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764928.308994:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764928.693078:0:10945:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1775764928.693102:0:10945:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:70783d78-a21c-41d0-8961-6627f7273664:10945:1862023066226176:192.168.202.105@tcp:400:kworker.0 00000100:00100000:1.0:1775764928.693263:0:10947:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:1.0:1775764928.693285:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066226432:192.168.202.105@tcp:400:kworker.0 00000100:00100000:3.0:1775764928.693350:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1775764928.693361:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066226560:192.168.202.105@tcp:400:kworker.0 00000100:00100000:2.0:1775764928.693389:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066226688:192.168.202.105@tcp:400:kworker.0 00010000:00010000:2.0:1775764929.333010:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764930.357015:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764931.380998:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764932.404999:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764933.429050:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764933.813132:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066226944:192.168.202.105@tcp:400:kworker.0 00000100:00100000:3.0:1775764933.813150:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190512300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066227072:192.168.202.105@tcp:400:kworker.0 00000100:00100000:2.0:1775764933.813184:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066227200:192.168.202.105@tcp:400:kworker.0 00010000:00010000:2.0:1775764934.453051:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764934.837113:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 1 seconds 00010000:00010000:2.0:1775764935.477022:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764935.860969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 1 seconds 00010000:00010000:2.0:1775764936.500974:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764936.884997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 1 seconds 00010000:00010000:2.0:1775764937.525032:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764937.909059:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 1 seconds 00010000:00010000:2.0:1775764938.549002:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764938.933023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 1 seconds 00000100:00100000:2.0:1775764938.933125:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190708380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066227584:192.168.202.105@tcp:400:kworker.0 00000100:00100000:1.0:1775764938.933135:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190708000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:70783d78-a21c-41d0-8961-6627f7273664:10946:1862023066227456:192.168.202.105@tcp:400:kworker.0 00000100:00100000:3.0:1775764938.933197:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066227712:192.168.202.105@tcp:400:kworker.0 00000100:00100000:2.0:1775764938.933238:0:10947:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00100000:2.0:1775764938.933250:0:10947:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066227968:192.168.202.105@tcp:400:kworker.0 00000100:00100000:3.0:1775764938.933259:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1775764938.933269:0:10944:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066227840:192.168.202.105@tcp:400:kworker.0 00010000:00010000:2.0:1775764939.572990:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1775764939.957026:0:10947:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764923/real 1775764923] req@ffff9d8190513480 x1862023066225280/t0(0) o4->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:6/4 lens 2064/448 e 0 to 1 dl 1775764939 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00000400:1.0:1775764939.957111:0:10945:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764923/real 1775764923] req@ffff9d8190708700 x1862023066225792/t0(0) o400->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1775764939.957111:0:10946:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764923/real 1775764923] req@ffff9d8190510e00 x1862023066225152/t0(0) o4->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:6/4 lens 2248/448 e 0 to 1 dl 1775764939 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1775764939.957147:0:10945:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8190708700 x1862023066225792/t0(0) o400->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764939 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775764939.957148:0:10946:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1775764939.957153:0:10946:0:(import.c:181:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff9d81892c2000: Connection to lustre-OST0000 (at 192.168.202.105@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1775764939.957156:0:10945:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1775764939.957159:0:10945:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9d81892c2000: Connection to lustre-MDT0000 (at 192.168.202.105@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1775764939.957166:0:10945:0:(import.c:416:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:1.0:1775764939.957181:0:10945:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764923/real 1775764923] req@ffff9d8190513100 x1862023066225024/t0(0) o4->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:6/4 lens 488/448 e 0 to 1 dl 1775764939 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1775764939.969731:0:10946:0:(import.c:416:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:1.0:1775764939.969748:0:10945:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e64800 already not connected (conn 1, was 1): DISCONN 00000100:00000400:3.0:1775764939.969758:0:10946:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764923/real 1775764923] req@ffff9d8190510a80 x1862023066224640/t0(0) o4->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:6/4 lens 488/448 e 0 to 1 dl 1775764939 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00000400:1.0:1775764939.969781:0:34105:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764923/real 1775764923] req@ffff9d8190511f80 x1862023066225536/t0(0) o35->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:23/10 lens 392/624 e 0 to 1 dl 1775764939 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1775764939.969785:0:10946:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e64800 already not connected (conn 1, was 1): DISCONN 00000100:00080000:1.0:1775764939.969813:0:34105:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:00100000:3.0:1775764939.969817:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190708000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:70783d78-a21c-41d0-8961-6627f7273664:10946:1862023066227456:192.168.202.105@tcp:400:kworker.0 00000100:02000400:1.0:1775764939.969820:0:34105:0:(import.c:181:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff9d81892c2000: Connection to lustre-MDT0001 (at 192.168.202.105@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1775764939.969826:0:34105:0:(import.c:416:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:1.0:1775764939.969835:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775764939.969903:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764939.969910:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775764939.969913:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775764939.969918:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1739 00000100:00080000:3.0:1775764939.969932:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775764939.969968:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066228096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775764939.969992:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764939.969999:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775764939.970004:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775764939.970013:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1744 00000100:00080000:3.0:1775764939.970039:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775764939.970049:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775764939.970067:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066228224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775764939.970084:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764939.970088:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775764939.970090:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775764939.970094:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1744 00000100:00080000:3.0:1775764939.970099:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775764939.970104:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775764939.970114:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066228352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775764939.970125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764939.970129:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775764939.970131:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775764939.970135:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1744 00000100:00080000:3.0:1775764939.970140:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775764939.970144:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775764939.970158:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066228480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775764939.970209:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00100000:3.0:1775764939.970219:0:10946:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066228608:192.168.202.105@tcp:400:kworker.0 00000100:00080000:2.0:1775764939.975380:0:10947:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e64800 already not connected (conn 2, was 1): CONNECTING 00010000:00010000:2.0:1775764940.597026:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764940.980994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764941.621015:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764942.004970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00000400:2.0:1775764942.004970:0:10947:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764925/real 1775764925] req@ffff9d819070bb80 x1862023066226048/t0(0) o16->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 440/432 e 0 to 1 dl 1775764941 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00000400:3.0:1775764942.004970:0:10946:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764925/real 1775764925] req@ffff9d819070a300 x1862023066225920/t0(0) o4->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:6/4 lens 5936/448 e 0 to 1 dl 1775764941 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1775764942.005038:0:10947:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1775764942.005043:0:10947:0:(import.c:181:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff9d81892c2000: Connection to lustre-OST0001 (at 192.168.202.105@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1775764942.021283:0:10946:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e62000 already not connected (conn 1, was 1): DISCONN 00000100:00080000:2.0:1775764942.021289:0:10947:0:(import.c:416:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00100000:3.0:1775764942.021496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066228736:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775764942.644962:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764943.029019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764943.669056:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764944.052976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00000400:2.0:1775764944.053023:0:10947:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764928/real 1775764928] req@ffff9d8190551880 x1862023066226432/t0(0) o400->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1775764944.053031:0:10944:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764928/real 1775764928] req@ffff9d81901af800 x1862023066226688/t0(0) o400->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764944.053091:0:10944:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d81901af800 x1862023066226688/t0(0) o400->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764944 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775764944.053112:0:10944:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e62000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1775764944.053119:0:10944:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764928/real 1775764928] req@ffff9d8190553480 x1862023066226304/t0(0) o400->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764944.053140:0:10944:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8190553480 x1862023066226304/t0(0) o400->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764944 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775764944.053146:0:10944:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d8190e60800 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:3.0:1775764944.053154:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066226688:192.168.202.105@tcp:400:kworker.0 00000100:00100000:3.0:1775764944.053160:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066226304:192.168.202.105@tcp:400:kworker.0 00000100:00100000:2.0:1775764944.063826:0:10947:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8190551880 x1862023066226432/t0(0) o400->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764944 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775764944.063836:0:10947:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d8190e66000 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:2.0:1775764944.063849:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066226432:192.168.202.105@tcp:400:kworker.0 00010000:00010000:2.0:1775764944.693069:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764945.076988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775764945.077113:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764945.077119:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764945.077122:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764945.077125:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764945.077128:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764945.077131:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764945.077133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764945.077136:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764945.077140:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764945.077143:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1775764945.717042:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764946.101220:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764946.740997:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764947.125004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775764947.125020:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764947.125026:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764947.125029:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764947.125032:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764947.125034:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764947.125037:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764947.125040:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764947.125042:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764947.125044:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764947.125046:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1775764947.765010:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775764948.149664:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764948.789017:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1775764949.109061:0:10945:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764933/real 1775764933] req@ffff9d8186fb8e00 x1862023066227328/t0(0) o400->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775764949.118471:0:10945:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8186fb8e00 x1862023066227328/t0(0) o400->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764949 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775764949.118484:0:10945:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e62000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1775764949.118500:0:10945:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10945:1862023066227328:192.168.202.105@tcp:400:kworker.0 00000100:00000400:3.0:1775764949.173342:0:10944:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764933/real 1775764933] req@ffff9d8190512d80 x1862023066227200/t0(0) o400->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764949.173385:0:10944:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8190512d80 x1862023066227200/t0(0) o400->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764949 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775764949.173396:0:10944:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e64800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:3.0:1775764949.173419:0:10944:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10944:1862023066227200:192.168.202.105@tcp:400:kworker.0 00000100:00000400:3.0:1775764949.173487:0:10946:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764933/real 1775764933] req@ffff9d8190510380 x1862023066226944/t0(0) o400->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764949.173506:0:10946:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8190510380 x1862023066226944/t0(0) o400->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764949 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775764949.173515:0:10946:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d8190e60800 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:3.0:1775764949.173527:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066226944:192.168.202.105@tcp:400:kworker.0 00000100:00080000:1.0:1775764949.173715:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764949.173729:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775764949.173734:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775764949.173742:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1776 00000100:00080000:1.0:1775764949.173755:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1775764949.173783:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00000400:2.0:1775764949.173862:0:10947:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764933/real 1775764933] req@ffff9d8190512300 x1862023066227072/t0(0) o400->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764949.173873:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:2.0:1775764949.173914:0:10947:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8190512300 x1862023066227072/t0(0) o400->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764949 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775764949.173923:0:10947:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d8190e66000 already not connected (conn 3, was 2): DISCONN 00000100:00100000:2.0:1775764949.173947:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190512300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066227072:192.168.202.105@tcp:400:kworker.0 00000100:00100000:1.0:1775764949.174412:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066228864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775764949.174448:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764949.174451:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775764949.174453:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775764949.174457:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1776 00000100:00080000:1.0:1775764949.174462:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1775764949.174468:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775764949.174479:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775764949.174491:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066228992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775764949.174499:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764949.174501:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775764949.174503:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775764949.174506:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1776 00000100:00080000:1.0:1775764949.174520:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1775764949.174522:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775764949.174525:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775764949.174534:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066229120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775764949.174540:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764949.174542:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775764949.174543:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775764949.174546:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1776 00000100:00080000:1.0:1775764949.174549:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1775764949.174551:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775764949.174555:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775764949.174563:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066229248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775764949.174572:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764949.174575:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764949.174590:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764949.174597:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1775764949.812992:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764950.196974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764950.837033:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764951.221031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764951.861021:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764952.245125:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775764952.245181:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764952.245186:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764952.245189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764952.245192:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764952.245194:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764952.245196:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764952.245198:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764952.245200:0:34107:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764952.245202:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764952.245205:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775764952.245208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775764952.245213:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1778 00000100:00080000:1.0:1775764952.245224:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1775764952.245236:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775764952.245252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775764952.245286:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066229376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1775764952.885134:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764953.268997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764953.909024:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764954.293012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00000400:2.0:1775764954.293020:0:10947:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764938/real 1775764938] req@ffff9d8188742300 x1862023066227968/t0(0) o400->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775764954.293086:0:16474:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764954.293094:0:16474:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764954.293100:0:16474:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764954.293106:0:16474:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764954.293122:0:16474:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764954.293126:0:16474:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764954.293127:0:16474:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764954.293129:0:16474:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764954.293131:0:16474:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764954.293133:0:16474:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1775764954.303225:0:10947:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8188742300 x1862023066227968/t0(0) o400->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764954 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775764954.303238:0:10947:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e62000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:2.0:1775764954.303245:0:10947:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764938/real 1775764938] req@ffff9d8190708380 x1862023066227584/t0(0) o400->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775764954.303253:0:10947:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8190708380 x1862023066227584/t0(0) o400->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 224/224 e 0 to 1 dl 1775764954 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775764954.303257:0:10947:0:(import.c:213:ptlrpc_set_import_discon()) mdc: import ffff9d8190e60800 already not connected (conn 4, was 2): CONNECTING 00000100:00100000:2.0:1775764954.303266:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066227968:192.168.202.105@tcp:400:kworker.0 00000100:00100000:2.0:1775764954.303270:0:10947:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190708380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10947:1862023066227584:192.168.202.105@tcp:400:kworker.0 00010000:00010000:2.0:1775764954.932995:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1775764955.317085:0:10946:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764939/real 1775764939] req@ffff9d8188742680 x1862023066228608/t0(0) o400->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764955.317116:0:10946:0:(client.c:2523:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9d8188742680 x1862023066228608/t0(0) o400->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 224/224 e 0 to 1 dl 1775764955 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764955.317134:0:10946:0:(import.c:213:ptlrpc_set_import_discon()) osc: import ffff9d8190e62000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1775764955.317144:0:10946:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10946:1862023066228608:192.168.202.105@tcp:400:kworker.0 00010000:00010000:2.0:1775764955.957013:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764956.980988:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764957.365058:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764958.004984:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764958.389020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764959.029031:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764959.412999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764960.052972:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764961.077047:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764961.461065:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764962.100993:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775764962.485005:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764962.485014:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764962.485018:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764962.485022:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764962.485024:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764962.485028:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764962.485033:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764962.485037:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764962.485040:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764962.485044:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1775764963.124974:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764964.148995:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764964.532998:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764949/real 1775764949] req@ffff9d8190551500 x1862023066228992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775764964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764964.533028:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066228992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775764964 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764964.533044:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764949/real 1775764949] req@ffff9d8190550e00 x1862023066229120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775764964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764964.533058:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066229120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775764964 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764964.533065:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764949/real 1775764949] req@ffff9d8190551880 x1862023066229248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775764964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764964.533078:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066229248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775764964 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764964.533090:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764949/real 1775764949] req@ffff9d8190553100 x1862023066228864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775764964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764964.533099:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066228864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775764964 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764964.533116:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764964.533136:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764964.533141:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066228992:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775764964.533148:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764964.533154:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764964.533159:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229120:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775764964.533166:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775764964.533175:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764964.533182:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764964.533184:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764964.533187:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764964.533193:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229248:192.168.202.105@tcp:8:kworker.0 00000100:00080000:3.0:1775764964.533194:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775764964.533198:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775764964.533201:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775764964.533205:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764964.533212:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066228864:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775764964.533213:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1785 00000100:00080000:3.0:1775764964.533229:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1775764964.533240:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775764964.533252:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775764964.533280:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066229504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775764964.533306:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764964.533310:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775764964.533312:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775764964.533317:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1785 00000100:00080000:3.0:1775764964.533322:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1775764964.533326:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775764964.533331:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775764964.533348:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066229632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764964.533359:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229504:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775764964.533363:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764964.533368:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775764964.533369:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775764964.533374:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1785 00000100:00080000:3.0:1775764964.533379:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1775764964.533383:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775764964.533389:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775764964.533401:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066229760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775764964.533412:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764964.533416:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764964.533424:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764964.533427:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775764964.533428:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775764964.533433:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1785 00000100:00080000:3.0:1775764964.533437:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1775764964.533441:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775764964.533456:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066229888/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775764964.533465:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764964.533470:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1775764964.533625:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764964.533668:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764964.533695:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066229888:192.168.202.105@tcp:250:kworker.0 00010000:00010000:2.0:1775764965.172983:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764966.196999:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764967.220987:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764967.604946:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764952/real 1775764952] req@ffff9d8190553480 x1862023066229376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775764967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764967.604977:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066229376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775764967 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764967.605000:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764967.605021:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764967.605026:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229376:192.168.202.105@tcp:8:kworker.0 00000100:00080000:3.0:1775764967.605062:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764967.605069:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764967.605072:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764967.605075:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764967.605078:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764967.605081:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764967.605084:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764967.605087:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1775764967.605089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764967.605092:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775764967.605096:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775764967.605102:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1788 00000100:00080000:3.0:1775764967.605112:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1775764967.605118:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775764967.605129:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775764967.605158:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066230016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764967.605214:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230016:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775764968.244980:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764969.268986:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764969.652972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775764969.653012:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764969.653017:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764969.653019:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764969.653023:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764969.653025:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764969.653027:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764969.653029:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764969.653031:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764969.653032:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764969.653034:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1775764970.293007:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764970.677020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764971.317020:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764971.700990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764972.341020:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775764972.661200:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764972.661215:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764972.661218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764972.661220:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764972.661222:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764972.661224:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764972.661226:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764972.661229:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764972.661231:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764972.661234:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1775764972.725003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764973.365002:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764973.749117:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764974.388998:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764974.772981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764975.412976:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764976.437039:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.1:1775764976.629008:0:10936:0:(socklnd_cb.c:2416:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.202.105@tcp (192.168.202.105:988) the network or that node may be down. 00000100:00020000:3.0:1775764976.629090:0:10936:0:(events.c:192:client_bulk_callback()) event type 1, status -5, req ffff9d8190513100 desc ffff9d8198485800 mbits 1862023066225024 00000100:00020000:3.0:1775764976.632252:0:10936:0:(events.c:192:client_bulk_callback()) event type 1, status -5, req ffff9d8190510a80 desc ffff9d8198487400 mbits 1862023066224640 00000100:00020000:3.0:1775764976.635138:0:10936:0:(events.c:192:client_bulk_callback()) event type 1, status -5, req ffff9d8190511180 desc ffff9d8198484400 mbits 1862023066224384 00010000:00010000:2.0:1775764977.461020:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:02000040:1.0:1775764977.653109:0:10941:0:(lib-move.c:3991:lnet_print_recovery_list()) 1 local NIs in recovery (showing 1): 192.168.202.5@tcp 00000100:00080000:1.0:1775764977.845000:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764977.845012:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764977.845016:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764977.845021:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764977.845024:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764977.845027:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764977.845031:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764977.845037:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1775764977.845039:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775764977.845041:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1775764978.484984:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764979.509005:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775764980.532999:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764980.917029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764981.556992:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764981.941007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764982.581012:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775764982.965024:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775764982.965034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775764982.965034:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764982.965037:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764982.965043:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764982.965046:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764982.965048:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764982.965049:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764982.965051:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764982.965053:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764982.965055:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1775764983.605000:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764983.988995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764984.629011:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764985.012964:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764964/real 1775764964] req@ffff9d8188741180 x1862023066229632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775764984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764985.012994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775764985.013006:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066229632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775764984 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764985.013018:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764964/real 1775764964] req@ffff9d8188741500 x1862023066229760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775764984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764985.013035:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066229760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775764984 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764985.013044:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764964/real 1775764964] req@ffff9d8188740a80 x1862023066229888/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775764984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764985.013054:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066229888/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775764984 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764985.013063:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764964/real 1775764964] req@ffff9d8188741f80 x1862023066229504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775764984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775764985.013074:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066229504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775764984 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775764985.013093:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775764985.013110:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775764985.013115:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229632:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775764985.013123:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775764985.013129:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775764985.013132:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229760:192.168.202.105@tcp:8:kworker.0 00000100:00080000:2.0:1775764985.013133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764985.013138:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764985.013140:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775764985.013143:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775764985.013144:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764985.013146:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764985.013153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775764985.013154:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775764985.013156:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764985.013161:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1801 00000100:00100000:3.0:1775764985.013161:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066229888:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775764985.013165:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775764985.013169:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775764985.013172:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066229504:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775764985.013173:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1775764985.013180:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764985.013194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775764985.013226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8186fb8e00 x1862023066230144/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764985.013249:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764985.013252:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775764985.013254:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764985.013258:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1801 00000100:00080000:2.0:1775764985.013262:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1775764985.013266:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764985.013271:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775764985.013276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775764985.013308:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066230272/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764985.013317:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764985.013320:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764985.013326:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764985.013329:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775764985.013330:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764985.013333:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1801 00000100:00080000:2.0:1775764985.013337:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1775764985.013340:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775764985.013350:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066230400/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764985.013357:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764985.013360:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775764985.013362:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764985.013366:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1801 00000100:00080000:2.0:1775764985.013369:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1775764985.013373:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764985.013378:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775764985.013392:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066230528/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775764985.013399:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230272:192.168.202.105@tcp:8:kworker.0 00000100:00080000:2.0:1775764985.013406:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764985.013410:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1775764985.013427:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066230400:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775764985.013458:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230528:192.168.202.105@tcp:38:kworker.0 00010000:00010000:2.0:1775764985.652976:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764986.037001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764986.676983:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775764987.061158:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775764967/real 1775764967] req@ffff9d8188743800 x1862023066230016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775764987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764987.061208:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066230016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775764987 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764987.061240:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764987.061265:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764987.061269:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230016:192.168.202.105@tcp:8:kworker.0 00000100:00080000:2.0:1775764987.061312:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764987.061319:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764987.061322:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764987.061325:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764987.061328:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764987.061331:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764987.061334:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764987.061337:0:1374:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1775764987.061339:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764987.061343:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775764987.061346:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764987.061353:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1804 00000100:00080000:2.0:1775764987.061365:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1775764987.061372:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764987.061385:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775764987.061422:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066230656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775764987.061459:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775764987.061519:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230656:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764987.125376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764987/real 1775764987] req@ffff9d81984a0700 x1862023066230656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764987.125415:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066230656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764987.125428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764987.125434:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764987.125438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230656:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764987.125448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764985/real 1775764987] req@ffff9d81984a3b80 x1862023066230528/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764987.125471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066230528/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764987.125480:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764987.125484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764987.125487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230528:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764987.125495:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764985/real 1775764987] req@ffff9d81984a3480 x1862023066230272/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764987.125508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066230272/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764987.125519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764987.125523:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764987.125526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230272:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764987.125534:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764985/real 1775764987] req@ffff9d81984a2d80 x1862023066230400/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764987.125548:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066230400/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764987.125556:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764987.125559:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764987.125563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066230400:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775764987.125571:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764985/real 1775764987] req@ffff9d8186fb8e00 x1862023066230144/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764987.125602:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8186fb8e00 x1862023066230144/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764987.125612:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764987.125616:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764987.125619:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8186fb8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230144:192.168.202.105@tcp:38:kworker.0 00010000:00010000:2.0:1775764987.701000:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764988.085106:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764988.724964:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764989.108977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764989.748987:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775764990.068965:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764990.068973:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775764990.068976:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764990.068983:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1821 00000100:00080000:2.0:1775764990.068991:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764990.068995:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775764990.069025:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066230784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764990.069046:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764990.069048:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775764990.069049:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764990.069052:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1821 00000100:00080000:2.0:1775764990.069054:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764990.069056:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764990.069062:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775764990.069069:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066230912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764990.069081:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764990.069085:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775764990.069088:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066230784:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775764990.069088:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764990.069099:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1821 00000100:00080000:2.0:1775764990.069102:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764990.069106:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764990.069111:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775764990.069131:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0380 x1862023066231040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764990.069141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764990.069144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775764990.069146:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764990.069149:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1821 00000100:00080000:2.0:1775764990.069153:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764990.069158:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764990.069162:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775764990.069172:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1c00 x1862023066231168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764990.069178:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764990.069181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775764990.069182:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764990.069185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1823 00000100:00080000:2.0:1775764990.069187:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764990.069189:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764990.069192:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775764990.069198:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775764990.069201:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066231296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764990.069236:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231040:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764990.069261:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231168:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764990.069293:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231296:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775764990.133003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775764990.197298:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.202.105:988 00000400:00000100:3.0:1775764990.197376:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775764990.197404:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775764990.197466:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775764990.197481:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775764990.197491:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775764990.197499:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775764990.198132:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764990/real 1775764990] req@ffff9d81984a2a00 x1862023066231296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764990.198173:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066231296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764990.198184:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764990.198189:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764990.198194:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231296:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764990.198202:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764990/real 1775764990] req@ffff9d81984a0000 x1862023066230912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764990.198219:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066230912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764990.198226:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764990.198230:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764990.198233:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066230912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764990.198240:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764990/real 1775764990] req@ffff9d81984a0380 x1862023066231040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764990.198251:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0380 x1862023066231040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764990.198257:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764990.198260:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764990.198263:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231040:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764990.198269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764990/real 1775764990] req@ffff9d81984a1c00 x1862023066231168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764990.198280:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1c00 x1862023066231168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764990.198287:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764990.198291:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764990.198294:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231168:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764990.198301:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764990/real 1775764990] req@ffff9d81984a2300 x1862023066230784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764990.198313:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066230784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764990.198319:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764990.198323:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764990.198325:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066230784:192.168.202.105@tcp:250:kworker.0 00010000:00010000:2.0:1775764990.772972:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764991.156974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764991.796972:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764992.180950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764992.820974:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764993.204995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764993.844992:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764994.229006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764994.868996:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775764995.252980:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764995.252987:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775764995.252991:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775764995.252997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775764995.252999:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1826 00000100:00080000:2.0:1775764995.253013:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764995.253020:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775764995.253050:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066231424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764995.253072:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764995.253075:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775764995.253076:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764995.253080:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1826 00000100:00080000:2.0:1775764995.253083:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764995.253086:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764995.253096:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775764995.253111:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066231552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764995.253124:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066231424:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775764995.253125:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764995.253137:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775764995.253139:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764995.253144:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1826 00000100:00080000:2.0:1775764995.253148:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764995.253153:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764995.253164:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775764995.253179:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066231680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764995.253189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764995.253192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775764995.253193:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764995.253195:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1826 00000100:00080000:2.0:1775764995.253198:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764995.253200:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764995.253203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775764995.253211:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066231808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775764995.253218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775764995.253221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775764995.253222:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775764995.253227:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1826 00000100:00080000:2.0:1775764995.253231:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775764995.253233:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775764995.253239:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775764995.253255:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066231936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775764995.253263:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764995.253314:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231680:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764995.253342:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231808:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764995.253367:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231936:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775764995.892977:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764996.277011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775764996.341144:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.202.105:988 00000400:00000100:3.0:1775764996.341177:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775764996.341193:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775764996.341224:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775764996.341233:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775764996.341241:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775764996.341248:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775764996.341259:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764995/real 1775764996] req@ffff9d81984a1f80 x1862023066231552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764996.341277:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066231552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764996.341285:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764996.341288:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764996.341291:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764996.341295:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764995/real 1775764996] req@ffff9d81984a1180 x1862023066231680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764996.341306:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066231680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764996.341312:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764996.341315:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764996.341318:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231680:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775764996.341324:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764995/real 1775764996] req@ffff9d81984a1500 x1862023066231808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764996.341334:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066231808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764996.341343:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764996.341346:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764996.341349:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231808:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775764996.341356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764995/real 1775764996] req@ffff9d81984a2680 x1862023066231424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764996.341366:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066231424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764996.341372:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764996.341375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764996.341377:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066231424:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775764996.341406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775764995/real 1775764996] req@ffff9d81984a0a80 x1862023066231936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775764996.341416:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066231936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775764996.341422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775764996.341425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775764996.341428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066231936:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775764996.916972:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764997.300982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764997.941040:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764998.324994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764998.965017:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775764999.349036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775764999.989018:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765000.373241:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765000.373375:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765000.373383:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765000.373388:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765000.373400:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1831 00000100:00080000:1.0:1775765000.373414:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765000.373422:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765000.373472:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066232064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765000.373497:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765000.373502:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765000.373503:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765000.373506:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1831 00000100:00080000:1.0:1775765000.373509:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765000.373512:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765000.373523:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765000.373525:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066232064:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765000.373534:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066232192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765000.373545:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765000.373548:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765000.373549:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765000.373552:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1831 00000100:00080000:1.0:1775765000.373554:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765000.373556:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765000.373559:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765000.373567:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066232320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765000.373572:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765000.373575:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765000.373576:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765000.373578:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1831 00000100:00080000:1.0:1775765000.373581:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765000.373583:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765000.373585:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765000.373592:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066232448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765000.373597:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765000.373600:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765000.373601:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765000.373603:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1831 00000100:00080000:1.0:1775765000.373605:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765000.373610:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765000.373613:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765000.373624:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066232576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765000.373643:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765000.373676:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232320:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765000.373692:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232448:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765000.373706:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232576:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765001.012998:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765001.396976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765002.037937:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765002.421190:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775765002.485259:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.202.105:988 00000400:00000100:2.0:1775765002.485307:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775765002.485327:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765002.485379:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765002.485389:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765002.485398:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765002.485412:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765002.485419:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765000/real 1775765002] req@ffff9d8190553100 x1862023066232192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765002.485456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066232192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765002.485466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765002.485470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765002.485474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765002.485482:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765000/real 1775765002] req@ffff9d8190551880 x1862023066232320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765002.485495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066232320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765002.485502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765002.485506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765002.485509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232320:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765002.485516:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765000/real 1775765002] req@ffff9d8190550e00 x1862023066232448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765002.485540:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066232448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765002.485549:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765002.485552:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765002.485556:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232448:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765002.485563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765000/real 1775765002] req@ffff9d8190551500 x1862023066232576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765002.485574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066232576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765002.485581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765002.485583:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765002.485587:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232576:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765002.485593:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765000/real 1775765002] req@ffff9d8190553480 x1862023066232064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765002.485605:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066232064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765002.485612:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765002.485620:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765002.485624:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066232064:192.168.202.105@tcp:250:kworker.0 00010000:00010000:2.0:1775765003.061032:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765003.445005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765004.085011:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765004.468991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765005.109020:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775765005.492993:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765005.493000:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775765005.493003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765005.493005:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765005.493020:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1837 00000100:00080000:1.0:1775765005.493030:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765005.493035:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765005.493065:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553800 x1862023066232704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765005.493087:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765005.493090:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765005.493092:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765005.493095:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1837 00000100:00080000:1.0:1775765005.493099:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765005.493104:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765005.493112:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765005.493127:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066232832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765005.493133:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066232704:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765005.493154:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765005.493158:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765005.493159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765005.493163:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1837 00000100:00080000:1.0:1775765005.493166:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765005.493169:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765005.493174:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765005.493184:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066232960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765005.493190:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765005.493192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765005.493193:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765005.493195:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1837 00000100:00080000:1.0:1775765005.493204:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765005.493207:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765005.493210:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765005.493216:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765005.493224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066233088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765005.493233:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765005.493236:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765005.493237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765005.493240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1837 00000100:00080000:1.0:1775765005.493242:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765005.493244:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765005.493248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765005.493254:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066233216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765005.493275:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765005/real 1775765005] req@ffff9d8190553800 x1862023066232704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765005.493292:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553800 x1862023066232704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765005.493300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765005.493304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765005.493308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066232704:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765005.493329:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765005.493344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765005.493351:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765005/real 1775765005] req@ffff9d8190553480 x1862023066232832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765005.493363:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066232832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765005.493368:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765005.493369:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765005.493371:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765005.493375:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232960:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765005.493384:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765005.493389:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765005/real 1775765005] req@ffff9d8190551500 x1862023066232960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765005.493396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066232960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765005.493400:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765005.493402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765005.493404:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066232960:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765005.493407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765005.493434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765005.493441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765005/real 1775765005] req@ffff9d8190550e00 x1862023066233088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765005.493448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066233088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765005.493464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765005.493466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765005.493468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233088:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765005.493472:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233216:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765005.493482:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765005.493487:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765005/real 1775765005] req@ffff9d8190551880 x1862023066233216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765005.493494:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066233216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765005.493499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765005.493500:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765005.493502:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233216:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765006.132992:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765006.516948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765007.156977:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765007.541006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765008.180949:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765008.564965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765009.204990:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765009.588997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765010.229003:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765010.613007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765010.613013:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765010.613026:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765010.613032:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765010.613045:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1842 00000100:00080000:1.0:1775765010.613060:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765010.613068:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765010.613124:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066233344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765010.613161:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765010.613166:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765010.613168:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765010.613173:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1842 00000100:00080000:1.0:1775765010.613178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765010.613183:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765010.613198:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765010.613220:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066233472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765010.613221:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066233344:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765010.613246:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765010.613251:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765010.613254:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765010.613260:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1842 00000100:00080000:1.0:1775765010.613265:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765010.613270:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765010.613276:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765010.613294:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066233600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765010.613305:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765010.613309:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765010.613311:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765010.613316:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1842 00000100:00080000:1.0:1775765010.613321:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765010.613327:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765010.613334:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765010.613353:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066233728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765010.613359:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233472:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775765010.613392:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765010.613396:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765010.613398:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765010.613405:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1842 00000100:00080000:1.0:1775765010.613412:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765010.613419:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765010.613419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233600:192.168.202.105@tcp:38:kworker.0 00000008:00100000:1.0:1775765010.613439:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765010.613461:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066233856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765010.613469:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233728:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765010.613514:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233856:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765011.252975:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765011.637007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765011.701387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765010/real 1775765011] req@ffff9d8190551500 x1862023066233728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765011.701421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066233728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765011.701431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765011.701435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765011.701438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233728:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765011.701444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765010/real 1775765011] req@ffff9d8190551880 x1862023066233472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765011.701456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066233472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765011.701464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765011.701468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765011.701471:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765011.701478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765010/real 1775765011] req@ffff9d8190550e00 x1862023066233600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765011.701493:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066233600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765011.701501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765011.701505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765011.701509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233600:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765011.701515:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765010/real 1775765011] req@ffff9d8190553100 x1862023066233344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765011.701526:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066233344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765011.701534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765011.701538:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765011.701541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066233344:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765011.701566:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765010/real 1775765011] req@ffff9d8190553480 x1862023066233856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765011.701580:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066233856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765011.701588:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765011.701592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765011.701595:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066233856:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765012.276974:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765012.661048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765013.300973:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765013.684984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765014.324993:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765014.709027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765015.348971:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775765015.733007:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765015.733022:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765015.733030:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765015.733045:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1847 00000100:00080000:1.0:1775765015.733065:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765015.733079:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765015.733112:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765015.733146:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553800 x1862023066233984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765015.733183:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765015.733187:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765015.733190:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765015.733194:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1847 00000100:00080000:1.0:1775765015.733198:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765015.733202:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765015.733215:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765015.733232:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066234112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765015.733241:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765015.733245:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765015.733248:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765015.733257:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1847 00000100:00080000:1.0:1775765015.733261:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765015.733266:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765015.733283:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765015.733296:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066234240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765015.733308:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765015.733312:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765015.733326:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765015.733331:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1847 00000100:00080000:1.0:1775765015.733336:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765015.733341:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765015.733347:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765015.733354:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066233984:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765015.733365:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066234368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765015.733379:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765015.733383:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765015.733386:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765015.733391:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1847 00000100:00080000:1.0:1775765015.733398:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765015.733403:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765015.733408:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765015.733423:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066234496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765015.733438:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765015.733511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765015/real 1775765015] req@ffff9d8190553800 x1862023066233984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765015.733533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553800 x1862023066233984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765015.733545:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765015.733551:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765015.733555:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066233984:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765015.733565:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765015.733588:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765015.733599:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765015/real 1775765015] req@ffff9d8190553480 x1862023066234112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765015.733620:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066234112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765015.733631:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765015.733634:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765015.733638:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765015.733644:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234240:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765015.733664:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765015.733675:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765015/real 1775765015] req@ffff9d8190553100 x1862023066234240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765015.733688:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066234240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765015.733697:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765015.733700:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765015.733704:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234240:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765015.733730:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765015.733751:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765015.733763:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765015/real 1775765015] req@ffff9d8190550e00 x1862023066234368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765015.733775:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066234368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765015.733785:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765015.733788:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765015.733792:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234368:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765015.733798:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234496:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765015.733819:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765015.733830:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765015/real 1775765015] req@ffff9d8190551880 x1862023066234496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765015.733857:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066234496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765015.733865:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765015.733869:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765015.733872:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234496:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765016.372977:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765016.756981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765017.396984:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765017.780990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765018.421003:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765018.805020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765019.445011:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765019.828998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765020.468996:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765020.853013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765020.853177:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066234624:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765020.853240:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765020.853297:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765020/real 1775765020] req@ffff9d81901ad180 x1862023066234624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765020.853317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066234624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765020.853340:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765020.853349:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765020.853356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066234624:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765020.853387:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765020.853414:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765020.853426:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765020/real 1775765020] req@ffff9d81901afb80 x1862023066234752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765020.853436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066234752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765020.853442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765020.853444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765020.853446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765020.853450:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234880:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765020.853460:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765020.853466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765020/real 1775765020] req@ffff9d81901af100 x1862023066234880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765020.853473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066234880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765020.853479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765020.853481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765020.853483:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066234880:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765020.853487:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765020.853497:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765020.853502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765020/real 1775765020] req@ffff9d81901ae680 x1862023066235008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765020.853509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066235008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765020.853525:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765020.853527:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765020.853530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235008:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765020.853540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235136:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765020.853551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765020.853558:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765020/real 1775765020] req@ffff9d81901aed80 x1862023066235136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765020.853565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066235136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765020.853570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765020.853572:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765020.853574:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235136:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765021.492997:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765021.877020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765022.516957:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765022.900959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765023.541016:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765023.924959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765024.564976:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765024.948955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765025.589016:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765025.973178:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066235264:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765025.973257:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000800:00000100:3.0:1775765025.973305:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765025.973392:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765025/real 1775765025] req@ffff9d81901aed80 x1862023066235264/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765025.973419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066235264/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765025.973432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765025.973437:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765025.973443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066235264:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765025.973471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765025.973495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765025.973505:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765025/real 1775765025] req@ffff9d81901ae680 x1862023066235392/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765025.973514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066235392/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765025.973520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765025.973523:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765025.973525:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765025.973529:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235520:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765025.973541:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765025.973548:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765025/real 1775765025] req@ffff9d81901af100 x1862023066235520/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765025.973556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066235520/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765025.973567:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765025.973569:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765025.973571:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235520:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765025.973574:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765025.973586:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765025.973591:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765025/real 1775765025] req@ffff9d81901afb80 x1862023066235648/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765025.973598:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066235648/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765025.973605:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765025.973607:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765025.973609:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235648:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765025.973612:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235776:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765025.973622:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765025.973627:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765025/real 1775765025] req@ffff9d81901ad180 x1862023066235776/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765025.973634:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066235776/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765025.973640:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765025.973642:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765025.973644:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066235776:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765026.613031:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765026.996943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765027.637047:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765028.020973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765028.660980:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765029.044947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765029.684997:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765030.069000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765030.709008:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765031.092989:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765031.093000:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775765031.093007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765031.093007:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765031.093020:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1862 00000100:00080000:2.0:1775765031.093033:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765031.093041:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765031.093128:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066235904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765031.093168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765031.093219:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765031.093221:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765031.093225:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1862 00000100:00080000:2.0:1775765031.093229:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765031.093233:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765031.093244:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765031.093257:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066236032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765031.093267:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765031.093270:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765031.093272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765031.093276:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1862 00000100:00080000:2.0:1775765031.093279:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765031.093283:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765031.093286:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765031.093300:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066236160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765031.093312:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066235904:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765031.093314:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765031.093331:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765031.093337:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765031.093342:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1862 00000100:00080000:2.0:1775765031.093345:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765031.093347:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765031.093352:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765031.093365:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066236288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765031.093373:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765031.093378:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765031.093380:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765031.093382:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1862 00000100:00080000:2.0:1775765031.093391:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765031.093395:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765031.093399:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765031.093409:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066236416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765031.093453:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765031.093492:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236160:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765031.093521:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236288:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765031.093547:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236416:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765031.733056:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765032.116988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765032.756978:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765033.140994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765033.205632:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765031/real 1775765033] req@ffff9d81984a3100 x1862023066236160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765033.205660:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066236160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765033.205673:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765033.205679:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765033.205683:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236160:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765033.205691:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765031/real 1775765033] req@ffff9d81984a0e00 x1862023066236288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765033.205705:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066236288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765033.205736:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765033.205740:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765033.205744:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236288:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765033.205751:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765031/real 1775765033] req@ffff9d81984a0a80 x1862023066236416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765033.205764:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066236416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765033.205772:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765033.205775:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765033.205779:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236416:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765033.206084:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765031/real 1775765033] req@ffff9d81984a3800 x1862023066235904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765033.206104:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066235904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765033.206114:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765033.206118:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765033.206121:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066235904:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765033.206128:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765031/real 1775765033] req@ffff9d81984a1880 x1862023066236032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765033.206168:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066236032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765033.206176:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765033.206180:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765033.206183:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236032:192.168.202.105@tcp:38:kworker.0 00010000:00010000:2.0:1775765033.781022:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765034.165059:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765034.804968:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765035.188967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765035.828986:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765036.212996:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765036.213006:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775765036.213007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765036.213011:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765036.213023:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1867 00000100:00080000:2.0:1775765036.213033:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765036.213041:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765036.213081:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066236544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765036.213110:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765036.213114:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765036.213116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765036.213119:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1867 00000100:00080000:2.0:1775765036.213122:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765036.213151:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765036.213161:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765036.213177:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066236672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765036.213187:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066236544:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765036.213203:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765036.213207:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765036.213209:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765036.213215:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1867 00000100:00080000:2.0:1775765036.213218:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765036.213222:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765036.213227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765036.213239:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066236800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765036.213247:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765036.213250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765036.213251:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765036.213254:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1867 00000100:00080000:2.0:1775765036.213264:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765036.213267:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765036.213272:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765036.213273:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765036.213286:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066236928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765036.213302:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765036.213305:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765036.213307:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765036.213313:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1867 00000100:00080000:2.0:1775765036.213318:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765036.213321:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765036.213327:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765036/real 1775765036] req@ffff9d81984a2680 x1862023066236544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775765036.213328:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765036.213344:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066237056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765036.213354:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066236544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765036.213374:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765036.213380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765036.213385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066236544:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765036.213405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765036.213425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765036.213445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765036/real 1775765036] req@ffff9d81984a1500 x1862023066236672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765036.213456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066236672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765036.213462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765036.213465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765036.213468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765036.213473:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236800:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765036.213485:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765036.213501:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765036/real 1775765036] req@ffff9d81984a1180 x1862023066236800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765036.213511:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066236800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765036.213518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765036.213530:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765036.213532:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236800:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765036.213537:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765036.213552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765036.213560:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765036/real 1775765036] req@ffff9d81984a1f80 x1862023066236928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765036.213570:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066236928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765036.213577:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765036.213579:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765036.213581:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066236928:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765036.213586:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237056:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765036.213621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765036.213629:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765036/real 1775765036] req@ffff9d81984a2300 x1862023066237056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765036.213638:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066237056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765036.213651:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765036.213653:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765036.213656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237056:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765036.853040:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765037.236968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765037.876979:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765038.260984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765038.900962:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765039.285227:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765039.925014:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765040.308962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765040.949008:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765041.333005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765041.333151:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066237184:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765041.333207:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765041.333248:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765041/real 1775765041] req@ffff9d81901af800 x1862023066237184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765041.333270:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066237184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765041.333284:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765041.333289:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765041.333295:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066237184:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765041.333314:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765041.333339:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765041.333354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765041/real 1775765041] req@ffff9d81901ad180 x1862023066237312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765041.333372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066237312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765041.333383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765041.333390:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765041.333395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765041.333402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237440:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765041.333416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765041.333423:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765041/real 1775765041] req@ffff9d81901afb80 x1862023066237440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765041.333430:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066237440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765041.333435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765041.333437:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765041.333439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237440:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765041.333447:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765041.333458:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765041.333463:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765041/real 1775765041] req@ffff9d81901af100 x1862023066237568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765041.333471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066237568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765041.333477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765041.333478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765041.333480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237568:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765041.333484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237696:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765041.333493:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765041.333499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765041/real 1775765041] req@ffff9d81901ae680 x1862023066237696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765041.333506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066237696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765041.333511:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765041.333513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765041.333515:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237696:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765041.973024:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765042.357932:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765042.996992:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765043.380953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765044.021012:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765044.404964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765045.045020:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765045.428949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765046.069051:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765046.452967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765046.453057:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765046.453068:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765046.453072:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765046.453081:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1877 00000100:00080000:2.0:1775765046.453090:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765046.453096:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765046.453137:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1c00 x1862023066237824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765046.453165:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765046.453168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765046.453169:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765046.453172:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1877 00000100:00080000:2.0:1775765046.453174:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765046.453176:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765046.453185:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765046.453195:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0380 x1862023066237952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765046.453203:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765046.453206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765046.453208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765046.453211:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1877 00000100:00080000:2.0:1775765046.453213:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765046.453216:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765046.453219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765046.453227:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066237824:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765046.453228:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066238080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765046.453241:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765046.453244:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765046.453245:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765046.453248:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1877 00000100:00080000:2.0:1775765046.453250:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765046.453253:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765046.453256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765046.453266:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066238208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765046.453272:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765046.453274:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765046.453276:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765046.453281:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1877 00000100:00080000:2.0:1775765046.453285:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765046.453290:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765046.453310:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775765046.453315:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765046.453337:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066238336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765046.453374:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765046/real 1775765046] req@ffff9d81984a1c00 x1862023066237824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765046.453400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1c00 x1862023066237824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765046.453415:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765046.453421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765046.453426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066237824:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765046.453451:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765046.453484:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765046.453497:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765046/real 1775765046] req@ffff9d81984a0380 x1862023066237952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765046.453520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0380 x1862023066237952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765046.453533:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765046.453536:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765046.453539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066237952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765046.453561:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238080:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765046.453577:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765046.453586:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765046/real 1775765046] req@ffff9d81984a0000 x1862023066238080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765046.453595:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066238080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765046.453602:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765046.453604:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765046.453607:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238080:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765046.453612:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765046.453625:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765046.453632:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765046/real 1775765046] req@ffff9d81984a2a00 x1862023066238208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765046.453641:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066238208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765046.453648:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765046.453650:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765046.453654:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238208:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765046.453659:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238336:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765046.453671:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765046.453677:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765046/real 1775765046] req@ffff9d81984a2d80 x1862023066238336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765046.453686:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066238336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765046.453693:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765046.453695:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765046.453697:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238336:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765047.093037:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765047.476944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765048.116971:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765048.500959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765049.140996:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765049.525049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765050.165052:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765050.548946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765051.189099:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765051.572964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765051.573190:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066238464:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765051.573282:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765051.573343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765051/real 1775765051] req@ffff9d81901aed80 x1862023066238464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765051.573362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066238464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765051.573373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765051.573377:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765051.573381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066238464:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765051.573405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765051.573425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765051.573435:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765051/real 1775765051] req@ffff9d81901ae680 x1862023066238592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765051.573445:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066238592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765051.573451:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765051.573453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765051.573456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765051.573460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238720:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765051.573473:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765051.573480:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765051/real 1775765051] req@ffff9d81901af100 x1862023066238720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765051.573489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066238720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765051.573495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765051.573498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765051.573500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238720:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765051.573504:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765051.573517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765051.573523:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765051/real 1775765051] req@ffff9d81901afb80 x1862023066238848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765051.573532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066238848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765051.573540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765051.573543:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765051.573545:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238848:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765051.573549:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238976:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765051.573560:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765051.573566:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765051/real 1775765051] req@ffff9d81901ad180 x1862023066238976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765051.573574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066238976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765051.573580:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765051.573582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765051.573584:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066238976:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765052.213009:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765052.597927:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765053.237035:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765053.621196:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765054.261050:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765054.645031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765055.284999:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765055.669686:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765056.309013:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765056.692971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765056.693161:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066239104:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765056.693229:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765056.693273:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765056/real 1775765056] req@ffff9d81901af800 x1862023066239104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765056.693289:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066239104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765056.693297:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765056.693300:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765056.693303:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066239104:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765056.693349:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765056.693366:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765056.693374:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765056/real 1775765056] req@ffff9d81901ad180 x1862023066239232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765056.693382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066239232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765056.693388:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765056.693390:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765056.693392:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765056.693396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239360:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765056.693432:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765056.693438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765056/real 1775765056] req@ffff9d81901afb80 x1862023066239360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765056.693445:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066239360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765056.693450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765056.693452:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765056.693453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239360:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765056.693457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765056.693466:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765056.693472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765056/real 1775765056] req@ffff9d81901af100 x1862023066239488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765056.693497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066239488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765056.693503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765056.693504:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765056.693506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239488:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765056.693509:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239616:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765056.693520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765056.693525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765056/real 1775765056] req@ffff9d81901ae680 x1862023066239616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765056.693533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066239616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765056.693538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765056.693539:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765056.693558:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239616:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765057.332995:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765057.717001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765058.357055:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765058.740968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765059.380976:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765059.765030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765060.404997:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765060.788972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765061.429020:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765061.812983:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765061.812991:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765061.812995:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765061.813006:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1893 00000100:00080000:2.0:1775765061.813015:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765061.813021:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765061.813053:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066239744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765061.813079:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765061.813081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765061.813083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765061.813086:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1893 00000100:00080000:2.0:1775765061.813092:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765061.813095:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765061.813102:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765061.813112:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066239872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765061.813119:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765061.813122:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765061.813123:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765061.813126:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1893 00000100:00080000:2.0:1775765061.813128:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765061.813130:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765061.813133:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765061.813139:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066240000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765061.813146:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765061.813149:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765061.813150:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765061.813154:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066239744:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765061.813155:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1893 00000100:00080000:2.0:1775765061.813159:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765061.813163:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765061.813166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765061.813174:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066240128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765061.813180:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765061.813182:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765061.813184:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765061.813186:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1893 00000100:00080000:2.0:1775765061.813189:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765061.813191:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765061.813194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765061.813199:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066240256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765061.813214:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765061.813250:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765061/real 1775765061] req@ffff9d81984a3480 x1862023066239744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765061.813263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066239744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765061.813271:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765061.813274:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765061.813278:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066239744:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765061.813282:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765061.813295:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765061.813302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765061/real 1775765061] req@ffff9d81984a3b80 x1862023066239872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765061.813310:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066239872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765061.813315:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765061.813317:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765061.813319:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066239872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765061.813334:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240000:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765061.813344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765061.813349:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765061/real 1775765061] req@ffff9d81984a0700 x1862023066240000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765061.813356:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066240000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765061.813362:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765061.813363:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765061.813366:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240000:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765061.813369:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765061.813379:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765061.813384:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765061/real 1775765061] req@ffff9d81984a2d80 x1862023066240128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765061.813391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066240128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765061.813396:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765061.813398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765061.813400:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240128:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765061.813403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240256:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765061.813411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765061.813416:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765061/real 1775765061] req@ffff9d81984a2a00 x1862023066240256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765061.813422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066240256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765061.813427:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765061.813429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765061.813431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240256:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765062.453037:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765063.477081:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765064.500975:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765065.524998:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765066.548971:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765066.933139:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765066.933280:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066240384:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765066.933414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765066.933450:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240640:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765066.933468:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240768:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765066.933488:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240896:192.168.202.105@tcp:8:kworker.0 00000400:00000100:1.0:1775765066.997177:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775765066.997218:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775765066.997233:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765066.997262:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765066.997268:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765066.997273:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765066.997278:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765066.997339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765066/real 1775765066] req@ffff9d81901ad180 x1862023066240896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765066.997366:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066240896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765066.997379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765066.997385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765066.997390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240896:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765066.997399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765066/real 1775765066] req@ffff9d81901ae680 x1862023066240512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765066.997412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066240512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765066.997419:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765066.997423:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765066.997426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765066.997433:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765066/real 1775765066] req@ffff9d81901af100 x1862023066240640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765066.997446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066240640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765066.997455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765066.997459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765066.997463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240640:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765066.997470:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765066/real 1775765066] req@ffff9d81901afb80 x1862023066240768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765066.997479:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066240768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765066.997487:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765066.997490:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765066.997493:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066240768:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765066.997499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765066/real 1775765066] req@ffff9d81901aed80 x1862023066240384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765066.997512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066240384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765066.997519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765066.997523:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765066.997526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066240384:192.168.202.105@tcp:250:kworker.0 00010000:00010000:2.0:1775765067.573013:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765067.956974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765068.597009:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765068.980930:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765069.621017:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765070.004968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765070.644982:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765071.028990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765071.669008:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765072.052984:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765072.052986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765072.052994:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765072.052998:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765072.053006:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1903 00000100:00080000:3.0:1775765072.053016:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765072.053022:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765072.053057:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066241024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765072.053082:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765072.053085:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765072.053086:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765072.053089:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1903 00000100:00080000:3.0:1775765072.053092:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765072.053095:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765072.053103:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765072.053112:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066241024:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765072.053114:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066241152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765072.053125:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765072.053127:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765072.053128:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765072.053131:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1903 00000100:00080000:3.0:1775765072.053133:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765072.053135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765072.053138:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765072.053146:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066241280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765072.053152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765072.053155:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765072.053156:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765072.053158:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1903 00000100:00080000:3.0:1775765072.053168:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765072.053170:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765072.053175:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765072.053176:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765072.053185:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066241408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765072.053193:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765072.053195:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765072.053196:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765072.053198:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1903 00000100:00080000:3.0:1775765072.053201:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765072.053203:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765072.053206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765072.053216:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066241536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765072.053218:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765072/real 1775765072] req@ffff9d8188741f80 x1862023066241024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765072.053254:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066241024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765072.053263:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765072.053267:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765072.053270:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066241024:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765072.053287:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765072.053301:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765072.053309:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765072/real 1775765072] req@ffff9d8188740a80 x1862023066241152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765072.053317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066241152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765072.053322:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765072.053324:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765072.053326:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765072.053330:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241280:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765072.053339:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765072.053344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765072/real 1775765072] req@ffff9d8188741500 x1862023066241280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765072.053351:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066241280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765072.053356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765072.053358:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765072.053359:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241280:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765072.053363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765072.053372:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765072.053378:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765072/real 1775765072] req@ffff9d8188741180 x1862023066241408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765072.053384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066241408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765072.053391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765072.053393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765072.053395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241408:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765072.053398:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241536:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765072.053407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765072.053412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765072/real 1775765072] req@ffff9d8188741880 x1862023066241536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765072.053418:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066241536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765072.053424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765072.053426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765072.053427:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241536:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765072.692977:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765073.076971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765073.717089:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765074.100969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765074.741030:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765075.124967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765075.765002:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765076.148960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765076.789014:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765077.172929:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765077.172937:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765077.172941:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765077.172943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765077.172951:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1908 00000100:00080000:3.0:1775765077.172960:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765077.172966:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765077.172996:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066241664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765077.173015:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765077.173018:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765077.173020:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765077.173022:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1908 00000100:00080000:3.0:1775765077.173028:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765077.173031:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765077.173039:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765077.173048:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066241664:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765077.173051:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066241792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765077.173061:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765077.173064:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765077.173064:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765077.173067:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1908 00000100:00080000:3.0:1775765077.173069:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765077.173071:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765077.173075:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765077.173082:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066241920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765077.173088:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765077.173090:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765077.173091:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765077.173094:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1908 00000100:00080000:3.0:1775765077.173107:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765077.173109:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765077.173113:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765077.173118:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765077.173123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066242048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765077.173130:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765077.173132:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765077.173134:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765077.173137:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1908 00000100:00080000:3.0:1775765077.173139:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765077.173141:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765077.173144:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765077.173153:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066242176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765077.173162:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765077/real 1775765077] req@ffff9d8188743100 x1862023066241664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765077.173175:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066241664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765077.173184:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765077.173187:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765077.173190:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066241664:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765077.173206:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765077.173224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765077.173231:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765077/real 1775765077] req@ffff9d8188740e00 x1862023066241792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765077.173239:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066241792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765077.173245:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765077.173247:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765077.173249:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765077.173253:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241920:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765077.173263:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765077.173268:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765077/real 1775765077] req@ffff9d8188743480 x1862023066241920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765077.173275:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066241920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765077.173280:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765077.173282:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765077.173285:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066241920:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765077.173288:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765077.173301:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765077.173311:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765077/real 1775765077] req@ffff9d8188742d80 x1862023066242048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765077.173323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066242048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765077.173332:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765077.173336:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765077.173339:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242048:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765077.173346:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242176:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765077.173375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765077.173387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765077/real 1775765077] req@ffff9d8188743b80 x1862023066242176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765077.173400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066242176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765077.173409:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765077.173412:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765077.173415:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242176:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765077.813031:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765078.196970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765078.837010:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765079.221143:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765079.860998:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765080.245307:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765080.885005:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765081.909015:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765082.292934:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765082.292944:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765082.292948:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765082.292958:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1913 00000100:00080000:3.0:1775765082.292969:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765082.292976:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765082.293012:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066242304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765082.293034:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765082.293040:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765082.293042:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765082.293046:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1913 00000100:00080000:3.0:1775765082.293051:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765082.293056:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765082.293066:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765082.293077:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066242304:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765082.293078:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066242432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765082.293089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765082.293092:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765082.293094:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765082.293098:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1913 00000100:00080000:3.0:1775765082.293101:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765082.293104:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765082.293108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765082.293118:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066242560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765082.293126:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765082.293129:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765082.293131:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765082.293134:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1913 00000800:00000100:1.0:1775765082.293136:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765082.293137:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765082.293141:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765082.293145:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765082.293154:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066242688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765082.293163:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765082.293166:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765082.293168:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765082.293171:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1913 00000100:00080000:3.0:1775765082.293174:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765082.293177:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765082/real 1775765082] req@ffff9d8188743b80 x1862023066242304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765082.293177:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765082.293182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765082.293190:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066242816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765082.293194:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066242304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765082.293205:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765082.293209:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765082.293213:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066242304:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765082.293234:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765082.293252:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765082.293271:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765082/real 1775765082] req@ffff9d8188742d80 x1862023066242432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765082.293281:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066242432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765082.293286:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765082.293288:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765082.293290:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765082.293295:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242560:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765082.293307:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765082.293312:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765082/real 1775765082] req@ffff9d8188743480 x1862023066242560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765082.293320:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066242560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765082.293325:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765082.293327:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765082.293329:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242560:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765082.293332:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765082.293343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765082.293348:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765082/real 1775765082] req@ffff9d8188740e00 x1862023066242688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765082.293355:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066242688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765082.293361:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765082.293363:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765082.293364:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242688:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765082.293368:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242816:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765082.293381:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765082.293391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765082/real 1775765082] req@ffff9d8188743100 x1862023066242816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765082.293403:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066242816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765082.293419:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765082.293422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765082.293426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066242816:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765082.933010:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765083.956993:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765084.981017:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765086.004980:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765087.029009:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765087.412966:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765087.412973:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765087.412977:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765087.412988:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1918 00000100:00080000:3.0:1775765087.412998:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765087.413004:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765087.413037:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066242944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765087.413057:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765087.413060:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765087.413061:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765087.413064:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1918 00000100:00080000:3.0:1775765087.413067:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765087.413070:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765087.413078:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765087.413087:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066243072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765087.413094:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765087.413096:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765087.413098:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765087.413102:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1918 00000100:00080000:3.0:1775765087.413103:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765087.413108:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765087.413113:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765087.413116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066242944:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765087.413121:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066243200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765087.413141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765087.413143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765087.413144:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765087.413147:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1918 00000100:00080000:3.0:1775765087.413149:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765087.413153:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765087.413158:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765087.413177:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066243328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765087.413188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765087.413190:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765087.413191:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765087.413193:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1918 00000100:00080000:3.0:1775765087.413202:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765087.413205:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765087.413211:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765087.413220:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066243456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765087.413224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765087.413312:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765087/real 1775765087] req@ffff9d8188741880 x1862023066242944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765087.413337:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066242944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765087.413351:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765087.413356:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765087.413361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066242944:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765087.413371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765087.413397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765087.413407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765087/real 1775765087] req@ffff9d8188741180 x1862023066243072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765087.413416:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066243072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765087.413423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765087.413426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765087.413428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765087.413452:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243200:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765087.413466:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765087.413473:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765087/real 1775765087] req@ffff9d8188741500 x1862023066243200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765087.413481:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066243200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765087.413486:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765087.413488:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765087.413490:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243200:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765087.413494:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765087.413506:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765087.413512:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765087/real 1775765087] req@ffff9d8188740a80 x1862023066243328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765087.413520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066243328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765087.413526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765087.413528:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765087.413530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243328:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765087.413533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243456:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765087.413552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765087.413561:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765087/real 1775765087] req@ffff9d8188741f80 x1862023066243456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765087.413573:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066243456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765087.413582:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765087.413586:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765087.413589:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243456:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765088.053012:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765089.077032:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765090.100986:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765091.124998:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765092.149018:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765092.532928:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765092.532938:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765092.532942:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765092.532950:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1924 00000100:00080000:3.0:1775765092.532959:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765092.532966:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765092.533011:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066243584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765092.533037:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765092.533041:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765092.533043:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765092.533047:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1924 00000100:00080000:3.0:1775765092.533051:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765092.533055:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765092.533066:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765092.533083:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066243712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765092.533093:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066243584:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765092.533095:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765092.533102:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765092.533104:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765092.533108:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1924 00000100:00080000:3.0:1775765092.533111:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765092.533113:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765092.533118:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765092.533128:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066243840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765092.533135:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765092.533137:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765092.533138:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765092.533141:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1924 00000100:00080000:3.0:1775765092.533158:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765092.533160:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765092.533163:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765092.533168:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765092.533173:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066243968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765092.533180:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765092.533183:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765092.533185:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765092.533188:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1924 00000100:00080000:3.0:1775765092.533190:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765092.533192:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765092.533195:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765092.533204:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066244096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765092.533212:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765092/real 1775765092] req@ffff9d8188743800 x1862023066243584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765092.533227:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066243584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765092.533235:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765092.533238:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765092.533241:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066243584:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765092.533265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765092.533282:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765092.533290:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765092/real 1775765092] req@ffff9d8188742680 x1862023066243712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765092.533298:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066243712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765092.533308:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765092.533310:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765092.533312:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765092.533316:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243840:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765092.533327:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765092.533332:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765092/real 1775765092] req@ffff9d8188742300 x1862023066243840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765092.533340:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066243840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765092.533345:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765092.533347:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765092.533349:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243840:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765092.533353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765092.533363:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765092.533368:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765092/real 1775765092] req@ffff9d8188740700 x1862023066243968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765092.533376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066243968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765092.533381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765092.533383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765092.533385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066243968:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765092.533389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244096:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765092.533399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765092.533404:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765092/real 1775765092] req@ffff9d8188741c00 x1862023066244096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765092.533410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066244096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765092.533416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765092.533417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765092.533419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244096:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765093.173060:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765094.196995:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765095.221035:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765096.244974:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765097.269095:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765097.652967:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765097.652979:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765097.652985:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765097.652996:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1929 00000100:00080000:3.0:1775765097.653015:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765097.653023:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765097.653069:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066244224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765097.653097:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765097.653100:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765097.653102:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765097.653105:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1929 00000100:00080000:3.0:1775765097.653107:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765097.653110:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765097.653123:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765097.653132:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066244352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765097.653144:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765097.653147:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765097.653148:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765097.653151:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1929 00000100:00080000:3.0:1775765097.653153:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765097.653155:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765097.653159:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765097.653171:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066244480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765097.653183:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765097.653190:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066244224:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765097.653190:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765097.653194:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765097.653211:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1929 00000100:00080000:3.0:1775765097.653218:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765097.653224:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765097.653231:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765097.653247:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066244608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765097.653255:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765097.653257:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765097.653259:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765097.653262:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1929 00000100:00080000:3.0:1775765097.653268:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775765097.653275:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765097.653279:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765097.653285:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765097.653297:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066244736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765097.653329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765097/real 1775765097] req@ffff9d8188740380 x1862023066244224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765097.653345:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066244224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765097.653353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765097.653357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765097.653360:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066244224:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765097.653367:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765097.653382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765097.653389:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765097/real 1775765097] req@ffff9d8188740000 x1862023066244352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765097.653397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066244352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765097.653402:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765097.653404:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765097.653406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765097.653423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244480:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765097.653434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765097.653439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765097/real 1775765097] req@ffff9d8188742a00 x1862023066244480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765097.653447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066244480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765097.653452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765097.653453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765097.653455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244480:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765097.653459:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765097.653468:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765097.653474:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765097/real 1775765097] req@ffff9d8188741c00 x1862023066244608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765097.653482:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066244608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765097.653487:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765097.653489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765097.653491:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244608:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765097.653494:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244736:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765097.653504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765097.653509:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765097/real 1775765097] req@ffff9d8188740700 x1862023066244736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765097.653516:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066244736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765097.653521:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765097.653523:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765097.653525:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244736:192.168.202.105@tcp:8:kworker.0 00010000:00010000:2.0:1775765098.293015:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765099.317105:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765100.341046:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765101.365044:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765102.389023:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765102.772985:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765102.772991:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765102.772995:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765102.773003:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1934 00000100:00080000:2.0:1775765102.773012:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765102.773019:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765102.773050:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066244864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765102.773074:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765102.773078:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765102.773080:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765102.773085:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1934 00000100:00080000:2.0:1775765102.773089:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765102.773094:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765102.773102:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066244864:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775765102.773104:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765102.773123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0380 x1862023066244992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765102.773136:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765102.773140:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765102.773141:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765102.773144:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1934 00000100:00080000:2.0:1775765102.773157:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775765102.773162:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765102.773167:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765102.773174:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765102.773191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1c00 x1862023066245120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765102.773200:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765102/real 1775765102] req@ffff9d81984a0000 x1862023066244864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765102.773234:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765102.773237:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775765102.773239:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066244864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765102.773240:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765102.773250:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1934 00000100:00080000:1.0:1775765102.773255:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765102.773256:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765102.773260:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765102.773263:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066244864:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765102.773275:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765102.773280:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765102.773286:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775765102.773293:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066245248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765102.773302:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765102.773305:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1775765102.773307:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765102.773308:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765102.773314:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1934 00000100:00080000:2.0:1775765102.773317:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765102.773322:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765102/real 1775765102] req@ffff9d81984a0380 x1862023066244992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765102.773322:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765102.773329:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765102.773342:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0380 x1862023066244992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765102.773344:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066245376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765102.773375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765102.773392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765102.773396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066244992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765102.773403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245120:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765102.773426:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765102.773439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765102/real 1775765102] req@ffff9d81984a1c00 x1862023066245120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765102.773453:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1c00 x1862023066245120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765102.773475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765102.773479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765102.773481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245120:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765102.773492:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765102.773525:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765102.773536:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765102/real 1775765102] req@ffff9d81984a2300 x1862023066245248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765102.773545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066245248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765102.773551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765102.773553:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765102.773555:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245248:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765102.773559:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245376:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765102.773572:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765102.773578:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765102/real 1775765102] req@ffff9d81984a1f80 x1862023066245376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765102.773585:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066245376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765102.773590:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765102.773592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765102.773595:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245376:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765103.413004:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765104.437207:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765105.461011:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765106.485143:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765107.509168:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765107.892993:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765107.893001:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765107.893005:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765107.893015:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1939 00000100:00080000:3.0:1775765107.893024:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765107.893032:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765107.893068:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066245504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765107.893093:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765107.893096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765107.893097:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765107.893100:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1939 00000100:00080000:3.0:1775765107.893103:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765107.893110:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765107.893118:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765107.893127:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066245632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765107.893135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765107.893139:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765107.893141:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765107.893146:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1939 00000100:00100000:1.0:1775765107.893148:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066245504:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765107.893148:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765107.893153:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765107.893160:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765107.893167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066245760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765107.893174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765107.893176:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765107.893177:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765107.893179:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1939 00000100:00080000:3.0:1775765107.893182:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765107.893185:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765107.893188:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765107.893195:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066245888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765107.893201:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765107.893204:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765107.893205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765107.893207:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1939 00000100:00080000:3.0:1775765107.893210:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775765107.893214:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765107.893217:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765107.893221:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765107.893229:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066246016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765107.893257:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765107/real 1775765107] req@ffff9d8188742300 x1862023066245504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765107.893272:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066245504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765107.893280:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765107.893283:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765107.893286:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066245504:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765107.893291:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765107.893306:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765107.893314:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765107/real 1775765107] req@ffff9d8188742680 x1862023066245632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765107.893322:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066245632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765107.893327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765107.893329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765107.893332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765107.893349:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245760:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765107.893361:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765107.893367:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765107/real 1775765107] req@ffff9d8188743800 x1862023066245760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765107.893374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066245760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765107.893380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765107.893383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765107.893386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245760:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765107.893393:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765107.893411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765107.893422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765107/real 1775765107] req@ffff9d8188741f80 x1862023066245888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765107.893436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066245888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765107.893446:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765107.893449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765107.893453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066245888:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765107.893488:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246016:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765107.893506:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765107.893514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765107/real 1775765107] req@ffff9d8188740a80 x1862023066246016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765107.893522:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066246016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765107.893528:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765107.893530:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765107.893532:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246016:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765108.533113:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765109.557042:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765110.581008:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765111.605062:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765112.629065:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765113.012953:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765113.012960:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765113.012964:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765113.012972:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1944 00000100:00080000:3.0:1775765113.012981:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765113.012987:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765113.013019:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066246144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765113.013043:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765113.013046:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765113.013047:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765113.013050:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1944 00000100:00080000:3.0:1775765113.013052:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765113.013054:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765113.013063:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765113.013080:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066246272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765113.013093:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765113.013097:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765113.013101:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765113.013108:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066246144:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765113.013110:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1944 00000100:00080000:3.0:1775765113.013125:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765113.013130:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765113.013135:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765113.013151:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066246400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765113.013160:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765113.013162:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765113.013165:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765113.013168:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1944 00000100:00080000:3.0:1775765113.013171:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765113.013173:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765113.013177:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765113.013184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066246528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765113.013190:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765113.013192:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765113.013195:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765113.013199:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1944 00000100:00080000:3.0:1775765113.013201:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765113.013203:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765113.013206:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765113.013214:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066246656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765113.013223:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765113.013306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765113/real 1775765113] req@ffff9d8188741500 x1862023066246144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765113.013329:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066246144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765113.013341:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765113.013346:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765113.013350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066246144:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765113.013380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765113.013406:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765113.013445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765113/real 1775765113] req@ffff9d8188741180 x1862023066246272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765113.013458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066246272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765113.013466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765113.013469:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765113.013473:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765113.013479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246400:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765113.013499:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765113.013508:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765113/real 1775765113] req@ffff9d8188741880 x1862023066246400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765113.013519:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066246400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765113.013526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765113.013529:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765113.013532:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246400:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765113.013537:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765113.013553:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765113.013561:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765113/real 1775765113] req@ffff9d8188743100 x1862023066246528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765113.013571:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066246528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765113.013581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765113.013584:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765113.013587:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246528:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765113.013592:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246656:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765113.013605:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765113.013612:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765113/real 1775765113] req@ffff9d8188740e00 x1862023066246656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765113.013622:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066246656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765113.013630:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765113.013632:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765113.013635:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246656:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765113.653066:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765114.677002:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765115.701004:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765116.725055:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765117.749099:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765118.132932:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765118.132941:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765118.132946:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765118.132955:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1949 00000100:00080000:3.0:1775765118.132965:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765118.132977:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765118.133012:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066246784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765118.133035:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765118.133038:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765118.133039:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765118.133042:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1949 00000100:00080000:3.0:1775765118.133044:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765118.133047:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765118.133054:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765118.133064:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066246912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765118.133073:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765118.133076:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765118.133080:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765118.133083:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066246784:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765118.133089:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1949 00000100:00080000:3.0:1775765118.133096:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765118.133101:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765118.133105:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765118.133118:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066247040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765118.133126:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765118.133129:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765118.133131:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765118.133134:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1949 00000100:00080000:3.0:1775765118.133137:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765118.133139:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765118.133144:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765118.133153:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066247168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765118.133160:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765118.133164:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765118.133166:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765118.133170:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1949 00000100:00080000:3.0:1775765118.133175:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775765118.133177:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765118.133189:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765118.133197:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765118.133216:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066247296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765118.133236:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765118/real 1775765118] req@ffff9d8188743480 x1862023066246784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765118.133254:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066246784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765118.133264:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765118.133268:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765118.133271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066246784:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765118.133293:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765118.133309:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765118.133317:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765118/real 1775765118] req@ffff9d8188742d80 x1862023066246912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765118.133329:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066246912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765118.133338:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765118.133341:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765118.133345:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066246912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765118.133351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247040:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765118.133386:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765118.133398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765118/real 1775765118] req@ffff9d8188743b80 x1862023066247040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765118.133410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066247040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765118.133416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765118.133419:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765118.133422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247040:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765118.133427:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765118.133443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765118.133450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765118/real 1775765118] req@ffff9d8188740e00 x1862023066247168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765118.133458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066247168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765118.133464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765118.133466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765118.133468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247168:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765118.133472:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247296:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765118.133482:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765118.133488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765118/real 1775765118] req@ffff9d8188743100 x1862023066247296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765118.133495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066247296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765118.133500:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765118.133502:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765118.133504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247296:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765118.773043:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765119.797091:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765120.821044:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765121.845104:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765122.869021:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765123.252946:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765123.252954:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765123.252960:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765123.252971:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1954 00000100:00080000:3.0:1775765123.252981:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765123.252989:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765123.253028:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066247424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765123.253054:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765123.253057:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765123.253058:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765123.253061:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1954 00000100:00080000:3.0:1775765123.253064:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765123.253067:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765123.253074:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765123.253082:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066247552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765123.253089:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765123.253091:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765123.253093:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765123.253096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1954 00000100:00080000:3.0:1775765123.253098:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765123.253100:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765123.253103:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765123.253109:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066247680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765123.253121:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765123.253125:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775765123.253126:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066247424:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765123.253127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765123.253131:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1954 00000100:00080000:3.0:1775765123.253135:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765123.253138:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765123.253141:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765123.253148:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066247808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765123.253153:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765123.253156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765123.253157:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765123.253159:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1954 00000100:00080000:3.0:1775765123.253161:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765123.253163:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765123.253166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765123.253172:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066247936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765123.253210:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765123.253258:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765123/real 1775765123] req@ffff9d8188741880 x1862023066247424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765123.253277:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066247424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765123.253288:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765123.253292:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765123.253297:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066247424:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765123.253305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765123.253361:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765123.253375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765123/real 1775765123] req@ffff9d8188741180 x1862023066247552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765123.253389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066247552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765123.253398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765123.253401:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765123.253405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765123.253426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247680:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765123.253449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765123.253461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765123/real 1775765123] req@ffff9d8188741500 x1862023066247680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765123.253474:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066247680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765123.253483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765123.253486:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765123.253490:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247680:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765123.253496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765123.253515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765123.253525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765123/real 1775765123] req@ffff9d8188740a80 x1862023066247808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765123.253537:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066247808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765123.253547:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765123.253551:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765123.253554:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247808:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765123.253560:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247936:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765123.253578:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765123.253587:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765123/real 1775765123] req@ffff9d8188741f80 x1862023066247936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765123.253607:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066247936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765123.253616:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765123.253620:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765123.253623:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066247936:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765123.893112:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765124.917079:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765125.941126:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765126.965045:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765127.989118:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765128.373000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765128.373150:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066248064:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765128.373337:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765128.373393:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248320:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765128.373438:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248448:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765128.373535:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248576:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765129.013008:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765129.396957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765130.037054:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765130.420949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765131.061033:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765131.444952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775765131.445104:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775765131.445142:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775765131.445156:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765131.445183:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765131.445189:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765131.445196:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765131.445202:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775765131.445222:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765128/real 1775765131] req@ffff9d81901ae680 x1862023066248576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765131.445280:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066248576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765131.445293:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765131.445298:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765131.445303:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248576:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765131.445311:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765128/real 1775765131] req@ffff9d81901af100 x1862023066248448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765131.445324:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066248448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765131.445333:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765131.445336:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765131.445338:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248448:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765131.445343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765128/real 1775765131] req@ffff9d81901aed80 x1862023066248192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765131.445351:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066248192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765131.445356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765131.445358:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765131.445360:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765131.445366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765128/real 1775765131] req@ffff9d81901afb80 x1862023066248320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765131.445378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066248320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765131.445386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765131.445389:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765131.445393:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248320:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765131.445399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765128/real 1775765131] req@ffff9d81901af800 x1862023066248064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765131.445411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066248064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765131.445418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765131.445422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765131.445425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066248064:192.168.202.105@tcp:250:kworker.0 00010000:00010000:1.0:1775765132.085030:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765132.468966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765133.109088:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765133.492974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765133.493073:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765133.493083:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765133.493087:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765133.493106:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1965 00000100:00080000:2.0:1775765133.493117:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765133.493125:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765133.493161:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066248704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765133.493188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765133.493192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765133.493194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765133.493198:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1965 00000100:00080000:2.0:1775765133.493202:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765133.493206:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765133.493217:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765133.493252:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066248704:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765133.493254:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066248832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765133.493280:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765133.493285:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765133.493287:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765133.493296:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1965 00000100:00080000:2.0:1775765133.493303:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765133.493309:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765133.493318:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765133.493350:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066248960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765133.493367:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765133.493383:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765133.493388:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765133.493391:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765133.493401:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1965 00000100:00080000:2.0:1775765133.493408:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765133.493416:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765133.493431:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765133.493447:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765133/real 1775765133] req@ffff9d81984a1180 x1862023066248704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765133.493458:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066249088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765133.493492:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765133.493501:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066248704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765133.493502:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765133.493509:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765133.493533:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1965 00000100:00080000:2.0:1775765133.493545:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765133.493549:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765133.493553:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765133.493566:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:1.0:1775765133.493568:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765133.493581:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066248704:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765133.493595:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066249216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765133.493622:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765133.493665:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765133.493684:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765133/real 1775765133] req@ffff9d81984a1500 x1862023066248832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765133.493696:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066248832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765133.493703:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765133.493708:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765133.493712:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765133.493720:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248960:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765133.493743:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765133.493756:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765133/real 1775765133] req@ffff9d81984a2680 x1862023066248960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765133.493770:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066248960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765133.493780:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765133.493784:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765133.493788:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066248960:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765133.493795:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765133.493819:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765133.493832:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765133/real 1775765133] req@ffff9d81984a1880 x1862023066249088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765133.493845:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066249088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765133.493855:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765133.493858:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765133.493862:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249088:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765133.493913:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249216:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765133.493936:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765133.493947:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765133/real 1775765133] req@ffff9d81984a3800 x1862023066249216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765133.493961:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066249216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765133.493970:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765133.493973:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765133.493977:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249216:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765134.133036:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765134.516979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765135.157113:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765135.540968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765136.181243:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765136.564997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765137.205208:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765137.589133:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765138.229117:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765138.612999:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765138.613009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765138.613011:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765138.613017:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765138.613028:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1970 00000100:00080000:2.0:1775765138.613060:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765138.613086:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765138.613152:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066249344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765138.613242:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765138.613247:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765138.613249:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765138.613254:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1970 00000100:00080000:2.0:1775765138.613258:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765138.613264:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765138.613276:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765138.613285:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066249344:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765138.613290:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066249472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765138.613300:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765138.613303:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765138.613304:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765138.613308:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1970 00000100:00080000:2.0:1775765138.613311:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765138.613314:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765138.613318:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765138.613329:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066249600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765138.613336:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765138.613338:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765138.613340:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765138.613342:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1970 00000100:00080000:2.0:1775765138.613360:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765138.613362:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765138.613365:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765138.613375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765138.613375:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066249728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765138.613384:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765138.613387:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765138.613388:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765138.613391:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1970 00000100:00080000:2.0:1775765138.613394:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765138.613397:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765138.613401:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765138.613409:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066249856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765138.613462:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765138/real 1775765138] req@ffff9d81984a0a80 x1862023066249344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765138.613496:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066249344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765138.613511:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765138.613517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765138.613523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066249344:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765138.613555:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765138.613586:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765138.613599:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765138/real 1775765138] req@ffff9d81984a0e00 x1862023066249472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765138.613614:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066249472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765138.613624:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765138.613628:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765138.613633:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765138.613640:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249600:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765138.613660:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765138.613669:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765138/real 1775765138] req@ffff9d81984a3100 x1862023066249600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765138.613682:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066249600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765138.613691:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765138.613694:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765138.613698:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249600:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765138.613703:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765138.613722:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765138.613731:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765138/real 1775765138] req@ffff9d81984a3800 x1862023066249728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765138.613744:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066249728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765138.613755:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765138.613758:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765138.613761:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249728:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765138.613782:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249856:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765138.613799:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765138.613811:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765138/real 1775765138] req@ffff9d81984a1880 x1862023066249856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765138.613824:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066249856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765138.613833:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765138.613837:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765138.613840:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066249856:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765139.253130:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765139.637035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765140.277070:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765140.660988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765141.301188:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765141.684949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765142.325101:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765142.708968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765143.349073:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765143.733003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765143.733017:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765143.733033:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765143.733038:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765143.733049:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1975 00000100:00080000:2.0:1775765143.733058:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765143.733063:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765143.733109:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066249984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765143.733186:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765143.733190:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765143.733192:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765143.733196:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1975 00000100:00080000:2.0:1775765143.733199:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765143.733202:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765143.733210:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765143.733219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066250112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765143.733230:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066249984:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765143.733230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765143.733248:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765143.733250:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765143.733254:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1975 00000100:00080000:2.0:1775765143.733256:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765143.733258:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765143.733261:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765143.733274:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066250240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765143.733284:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765143.733288:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765143.733290:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765143.733294:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1975 00000100:00080000:2.0:1775765143.733320:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765143.733325:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765143.733331:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765143.733333:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765143.733355:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066250368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765143.733365:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765143.733368:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765143.733370:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765143.733373:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1975 00000100:00080000:2.0:1775765143.733377:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765143.733381:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765143.733387:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765143.733399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765143/real 1775765143] req@ffff9d81984a2680 x1862023066249984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765143.733400:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066250496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765143.733488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066249984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765143.733502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765143.733507:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765143.733512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066249984:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765143.733537:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765143.733559:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765143.733569:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765143/real 1775765143] req@ffff9d81984a1500 x1862023066250112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765143.733579:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066250112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765143.733586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765143.733589:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765143.733592:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765143.733596:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250240:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765143.733609:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765143.733616:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765143/real 1775765143] req@ffff9d81984a1180 x1862023066250240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765143.733625:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066250240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765143.733631:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765143.733634:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765143.733636:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250240:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765143.733641:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765143.733654:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765143.733662:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765143/real 1775765143] req@ffff9d81984a1f80 x1862023066250368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765143.733671:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066250368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765143.733679:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765143.733681:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765143.733684:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250368:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765143.733688:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250496:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765143.733701:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765143.733708:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765143/real 1775765143] req@ffff9d81984a2300 x1862023066250496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765143.733717:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066250496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765143.733724:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765143.733726:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765143.733728:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250496:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765144.373047:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765144.756984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765145.397162:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765145.780951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765146.421204:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765146.804957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765147.445196:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765147.828964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765148.469199:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765148.853037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765148.853389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066250624:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775765148.853488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765148.853559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765148/real 1775765148] req@ffff9d81901ad180 x1862023066250624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765148.853589:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066250624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765148.853613:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765148.853625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765148.853635:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066250624:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765148.853680:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765148.853723:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765148.853741:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765148/real 1775765148] req@ffff9d81901af800 x1862023066250752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765148.853756:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066250752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765148.853766:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765148.853770:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765148.853774:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765148.853780:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250880:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765148.853815:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765148.853827:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765148/real 1775765148] req@ffff9d81901afb80 x1862023066250880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765148.853841:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066250880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765148.853850:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765148.853853:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765148.853857:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066250880:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765148.853863:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765148.853920:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765148.853932:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765148/real 1775765148] req@ffff9d81901aed80 x1862023066251008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765148.853945:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066251008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765148.853955:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765148.853959:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765148.853962:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251008:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765148.853978:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251136:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765148.853997:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765148.854007:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765148/real 1775765148] req@ffff9d81901af100 x1862023066251136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765148.854019:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066251136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765148.854028:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765148.854031:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765148.854035:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251136:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765149.493054:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765149.876957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765150.517099:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765150.901000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765151.541138:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765151.925016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765152.564999:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765152.949017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765153.589121:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765153.973009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765153.973221:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066251264:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775765153.973311:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765153.973405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765153/real 1775765153] req@ffff9d81901ae680 x1862023066251264/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765153.973497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066251264/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765153.973518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765153.973527:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765153.973534:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066251264:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765153.973570:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765153.973617:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765153.973642:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765153/real 1775765153] req@ffff9d81901af100 x1862023066251392/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765153.973664:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066251392/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765153.973675:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765153.973679:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765153.973683:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765153.973690:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251520:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765153.973717:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765153.973729:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765153/real 1775765153] req@ffff9d81901aed80 x1862023066251520/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765153.973744:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066251520/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765153.973756:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765153.973761:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765153.973765:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251520:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765153.973774:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765153.973804:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765153.973814:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765153/real 1775765153] req@ffff9d81901afb80 x1862023066251648/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765153.973828:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066251648/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765153.973840:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765153.973844:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765153.973848:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251648:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765153.973854:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251776:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765153.973871:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765153.973916:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765153/real 1775765153] req@ffff9d81901af800 x1862023066251776/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765153.973929:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066251776/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765153.973939:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765153.973943:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765153.973946:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066251776:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765154.613132:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765154.997064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765155.637073:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765156.020966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765156.661260:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765157.044951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765157.685154:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765158.069039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765158.709059:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765159.092941:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765159.092949:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765159.092954:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765159.092963:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1990 00000100:00100000:1.0:1775765159.092973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765159.092984:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765159.093006:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765159.093043:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066251904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765159.093063:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765159.093066:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765159.093067:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765159.093070:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1990 00000100:00080000:3.0:1775765159.093073:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765159.093076:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765159.093084:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765159.093093:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066252032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765159.093098:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066251904:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765159.093110:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765159.093114:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765159.093116:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765159.093122:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1990 00000100:00080000:3.0:1775765159.093125:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765159.093136:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765159.093145:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765159.093158:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066252160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765159.093166:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765159.093176:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765159.093179:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765159.093181:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765159.093185:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1990 00000100:00080000:3.0:1775765159.093188:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765159.093192:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765159.093198:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765159.093209:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765159/real 1775765159] req@ffff9d8188743800 x1862023066251904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765159.093213:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066252288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765159.093251:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066251904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765159.093252:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765159.093262:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765159.093265:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765159.093269:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765159.093270:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1990 00000100:00080000:1.0:1775765159.093277:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775765159.093277:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765159.093282:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066251904:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765159.093286:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765159.093294:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765159.093305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765159.093311:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066252416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765159.093327:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765159.093337:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765159/real 1775765159] req@ffff9d8188742680 x1862023066252032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765159.093346:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066252032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765159.093352:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765159.093354:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765159.093356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765159.093360:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252160:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765159.093370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765159.093376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765159/real 1775765159] req@ffff9d8188742300 x1862023066252160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765159.093388:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066252160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765159.093393:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765159.093395:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765159.093396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252160:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765159.093400:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765159.093410:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765159.093415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765159/real 1775765159] req@ffff9d8188740700 x1862023066252288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765159.093423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066252288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765159.093429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765159.093430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765159.093432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252288:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765159.093440:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252416:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765159.093450:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765159.093455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765159/real 1775765159] req@ffff9d8188741c00 x1862023066252416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765159.093462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066252416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765159.093471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765159.093473:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765159.093475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252416:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765159.733038:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765160.116931:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765160.757124:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765161.141031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765161.781132:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765162.164971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765162.805147:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765163.189064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765163.829119:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765164.212985:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765164.212991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765164.212998:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765164.213003:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765164.213013:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 1995 00000100:00080000:3.0:1775765164.213022:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765164.213029:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765164.213061:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066252544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765164.213084:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765164.213088:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765164.213089:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765164.213092:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1995 00000100:00080000:3.0:1775765164.213095:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765164.213098:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765164.213105:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765164.213116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066252544:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765164.213117:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066252672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765164.213131:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765164.213133:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765164.213134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765164.213138:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1995 00000100:00080000:3.0:1775765164.213140:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765164.213143:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765164.213147:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765164.213156:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066252800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765164.213163:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765164.213166:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765164.213168:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765164.213170:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1995 00000100:00080000:3.0:1775765164.213174:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775765164.213178:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765164.213182:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765164.213187:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765164.213197:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066252928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765164.213204:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765164.213206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765164.213207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765164.213210:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 1995 00000100:00080000:3.0:1775765164.213212:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765164.213214:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765164.213219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765164.213248:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765164/real 1775765164] req@ffff9d8188742a00 x1862023066252544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765164.213258:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066253056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765164.213287:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066252544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765164.213302:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765164.213308:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765164.213314:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066252544:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765164.213337:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765164.213359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765164.213369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765164/real 1775765164] req@ffff9d8188740000 x1862023066252672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765164.213379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066252672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765164.213386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765164.213388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765164.213391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765164.213395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252800:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765164.213407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765164.213414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765164/real 1775765164] req@ffff9d8188740380 x1862023066252800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765164.213442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066252800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765164.213449:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765164.213452:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765164.213455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252800:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765164.213460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765164.213477:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765164.213486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765164/real 1775765164] req@ffff9d8188741c00 x1862023066252928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765164.213497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066252928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765164.213506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765164.213509:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765164.213513:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066252928:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765164.213517:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253056:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765164.213531:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765164.213543:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765164/real 1775765164] req@ffff9d8188740700 x1862023066253056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765164.213562:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066253056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765164.213574:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765164.213579:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765164.213584:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253056:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765164.853228:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765165.236973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765165.877066:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765166.260962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765166.901114:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765167.285008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765167.925084:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765168.308997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765168.949076:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765169.332984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765169.332988:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765169.332998:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765169.333003:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765169.333022:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2000 00000100:00080000:3.0:1775765169.333033:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765169.333041:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765169.333111:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066253184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765169.333159:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765169.333167:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765169.333169:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765169.333174:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2000 00000100:00080000:3.0:1775765169.333180:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765169.333186:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765169.333198:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765169.333205:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066253184:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765169.333222:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066253312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765169.333233:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765169.333236:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765169.333238:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765169.333241:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2000 00000100:00080000:3.0:1775765169.333245:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765169.333248:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765169.333252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765169.333262:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066253440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765169.333271:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765169.333274:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765169.333275:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765169.333278:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2000 00000100:00080000:3.0:1775765169.333281:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765169.333286:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765169.333291:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765169.333294:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765169.333304:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066253568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765169.333313:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765169.333316:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765169.333317:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765169.333320:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2000 00000100:00080000:3.0:1775765169.333322:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765169.333325:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765169.333329:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765169.333339:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066253696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765169.333352:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765169/real 1775765169] req@ffff9d8188742300 x1862023066253184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765169.333371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066253184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765169.333382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765169.333386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765169.333389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066253184:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765169.333409:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765169.333441:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765169.333450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765169/real 1775765169] req@ffff9d8188742680 x1862023066253312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765169.333460:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066253312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765169.333467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765169.333470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765169.333472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765169.333477:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253440:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765169.333488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765169.333495:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765169/real 1775765169] req@ffff9d8188743800 x1862023066253440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765169.333509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066253440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765169.333516:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765169.333518:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765169.333520:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253440:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765169.333524:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765169.333536:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765169.333550:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765169/real 1775765169] req@ffff9d8188741f80 x1862023066253568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765169.333564:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066253568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765169.333587:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765169.333591:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765169.333596:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253568:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765169.333604:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253696:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765169.333629:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765169.333640:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765169/real 1775765169] req@ffff9d8188740a80 x1862023066253696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765169.333650:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066253696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765169.333676:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765169.333679:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765169.333682:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253696:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765169.973066:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765170.357020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765170.997072:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765172.021089:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765173.045027:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765174.069054:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765174.452949:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765174.452962:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765174.452968:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765174.452981:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2005 00000100:00080000:3.0:1775765174.452996:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765174.453005:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765174.453062:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066253824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765174.453092:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765174.453095:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765174.453097:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765174.453101:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2005 00000100:00080000:3.0:1775765174.453105:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765174.453109:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765174.453120:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765174.453133:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066253952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765174.453143:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765174.453147:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765174.453148:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765174.453152:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2005 00000100:00080000:3.0:1775765174.453155:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765174.453159:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765174.453163:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765174.453173:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066254080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765174.453179:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066253824:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765174.453188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765174.453192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765174.453194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765174.453199:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2005 00000100:00080000:3.0:1775765174.453201:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765174.453205:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765174.453210:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765174.453222:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066254208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765174.453254:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765174.453258:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765174.453260:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775765174.453268:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765174.453273:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2005 00000100:00080000:3.0:1775765174.453282:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765174.453289:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765174.453296:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765174.453316:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066254336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765174.453334:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765174/real 1775765174] req@ffff9d8188741500 x1862023066253824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765174.453399:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066253824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765174.453409:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765174.453413:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765174.453417:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066253824:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765174.453424:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765174.453441:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765174.453449:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765174/real 1775765174] req@ffff9d8188741180 x1862023066253952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765174.453466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066253952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765174.453474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765174.453478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765174.453481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066253952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765174.453506:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254080:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765174.453529:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765174.453547:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765174/real 1775765174] req@ffff9d8188741880 x1862023066254080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765174.453555:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066254080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765174.453561:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765174.453563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765174.453565:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254080:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765174.453570:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765174.453583:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765174.453588:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765174/real 1775765174] req@ffff9d8188743100 x1862023066254208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765174.453595:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066254208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765174.453606:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765174.453608:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765174.453610:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254208:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765174.453613:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254336:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765174.453632:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765174.453641:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765174/real 1775765174] req@ffff9d8188740e00 x1862023066254336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765174.453649:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066254336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765174.453654:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765174.453656:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765174.453658:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254336:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765175.093082:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765176.117072:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765177.141040:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765178.165085:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765179.189080:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765179.572987:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765179.573010:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765179.573017:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765179.573031:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2011 00000100:00080000:3.0:1775765179.573045:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765179.573055:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765179.573125:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066254464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765179.573165:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765179.573170:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765179.573172:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765179.573177:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2011 00000100:00080000:3.0:1775765179.573192:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765179.573210:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765179.573225:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765179.573241:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066254592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765179.573254:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765179.573259:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765179.573261:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765179.573266:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066254464:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765179.573269:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2011 00000100:00080000:3.0:1775765179.573277:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765179.573282:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765179.573289:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765179.573303:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066254720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765179.573312:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765179.573315:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765179.573317:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765179.573320:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2011 00000100:00080000:3.0:1775765179.573336:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765179.573340:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765179.573344:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765179.573350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765179.573357:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066254848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765179.573365:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765179.573368:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765179.573370:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765179.573374:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2011 00000100:00080000:3.0:1775765179.573377:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765179.573379:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765179.573383:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765179.573392:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066254976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765179.573414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765179/real 1775765179] req@ffff9d8188743b80 x1862023066254464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765179.573433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066254464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765179.573444:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765179.573449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765179.573455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066254464:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765179.573485:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765179.573512:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765179.573526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765179/real 1775765179] req@ffff9d8188742d80 x1862023066254592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765179.573541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066254592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765179.573549:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765179.573553:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765179.573556:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765179.573562:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254720:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765179.573585:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765179.573597:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765179/real 1775765179] req@ffff9d8188743480 x1862023066254720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765179.573611:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066254720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765179.573617:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765179.573619:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765179.573622:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254720:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765179.573627:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765179.573644:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765179.573650:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765179/real 1775765179] req@ffff9d8188740e00 x1862023066254848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765179.573658:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066254848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765179.573665:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765179.573668:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765179.573670:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254848:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765179.573673:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254976:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765179.573683:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765179.573689:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765179/real 1775765179] req@ffff9d8188743100 x1862023066254976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765179.573696:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066254976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765179.573701:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765179.573703:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765179.573705:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066254976:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765180.213078:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765181.237118:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765182.261062:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765183.285053:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765184.309056:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765184.693050:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765184.693068:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765184.693076:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765184.693093:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2016 00000100:00080000:3.0:1775765184.693109:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765184.693119:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765184.693167:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066255104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765184.693208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765184.693215:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765184.693219:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765184.693227:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2016 00000100:00080000:3.0:1775765184.693233:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765184.693240:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765184.693256:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765184.693283:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066255232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765184.693301:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066255104:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765184.693337:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765184.693341:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765184.693344:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765184.693353:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2016 00000100:00080000:3.0:1775765184.693359:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765184.693365:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765184.693374:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765184.693394:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066255360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765184.693407:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775765184.693416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765184.693418:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765184.693423:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765184.693431:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2016 00000100:00080000:3.0:1775765184.693439:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765184.693445:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765184.693453:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765184.693474:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066255488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765184.693487:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765184.693492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765184/real 1775765184] req@ffff9d8188741880 x1862023066255104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765184.693494:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765184.693500:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765184.693515:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2016 00000100:00080000:3.0:1775765184.693524:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765184.693530:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765184.693536:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765184.693537:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066255104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765184.693558:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775765184.693560:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066255616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765184.693569:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765184.693584:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066255104:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765184.693625:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765184.693655:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765184.693670:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765184/real 1775765184] req@ffff9d8188741180 x1862023066255232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765184.693685:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066255232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765184.693696:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765184.693700:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765184.693704:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765184.693711:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255360:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765184.693733:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765184.693743:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765184/real 1775765184] req@ffff9d8188741500 x1862023066255360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765184.693786:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066255360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765184.693797:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765184.693801:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765184.693804:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255360:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765184.693812:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765184.693838:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765184.693851:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765184/real 1775765184] req@ffff9d8188740a80 x1862023066255488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765184.693865:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066255488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765184.693915:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765184.693920:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765184.693924:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255488:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765184.693931:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255616:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765184.693955:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765184.693968:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765184/real 1775765184] req@ffff9d8188741f80 x1862023066255616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765184.693992:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066255616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765184.694003:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765184.694009:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765184.694015:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255616:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765185.333152:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765186.357123:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765187.381024:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765188.405037:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765189.429044:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765189.813021:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765189.813037:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765189.813044:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765189.813073:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2021 00000100:00080000:3.0:1775765189.813090:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765189.813101:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765189.813163:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066255744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765189.813209:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765189.813217:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765189.813221:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765189.813228:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2021 00000100:00080000:3.0:1775765189.813235:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765189.813243:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765189.813260:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765189.813291:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066255872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765189.813308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066255744:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765189.813323:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765189.813335:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765189.813338:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765189.813348:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2021 00000100:00080000:3.0:1775765189.813356:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765189.813364:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765189.813374:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765189.813400:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066256000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765189.813417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765189.813429:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765189.813436:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765189.813441:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765189.813453:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2021 00000100:00080000:3.0:1775765189.813462:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765189.813471:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765189.813482:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765189.813494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765189/real 1775765189] req@ffff9d8188743800 x1862023066255744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765189.813512:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066256128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765189.813527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066255744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765189.813531:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765189.813548:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765189.813555:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765189.813569:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765189.813569:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2021 00000100:00080000:3.0:1775765189.813584:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765189.813587:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775765189.813594:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765189.813600:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066255744:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775765189.813608:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765189.813629:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066256256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765189.813652:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765189.813754:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765189.813782:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765189/real 1775765189] req@ffff9d8188742680 x1862023066255872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765189.813801:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066255872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765189.813812:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765189.813816:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765189.813821:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066255872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765189.813829:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256000:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765189.813856:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765189.813868:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765189/real 1775765189] req@ffff9d8188742300 x1862023066256000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765189.813927:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066256000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765189.813937:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765189.813942:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765189.813946:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256000:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765189.813954:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765189.813978:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765189.813991:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765189/real 1775765189] req@ffff9d8188740700 x1862023066256128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765189.814005:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066256128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765189.814024:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765189.814028:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765189.814032:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256128:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765189.814051:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256256:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765189.814077:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765189.814095:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765189/real 1775765189] req@ffff9d8188741c00 x1862023066256256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765189.814119:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066256256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765189.814136:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765189.814143:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765189.814150:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256256:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765190.453047:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765191.477181:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765192.501166:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765193.525040:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765194.549099:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765194.932987:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765194.932993:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765194.932996:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765194.933004:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2026 00000100:00080000:3.0:1775765194.933014:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765194.933019:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765194.933053:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066256384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765194.933077:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765194.933080:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765194.933081:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765194.933084:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2026 00000100:00080000:3.0:1775765194.933086:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765194.933088:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765194.933095:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765194.933102:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066256512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765194.933109:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765194.933111:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765194.933112:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765194.933114:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2026 00000100:00080000:3.0:1775765194.933117:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765194.933119:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765194.933122:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765194.933127:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066256640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765194.933132:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765194.933134:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765194.933135:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765194.933138:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2026 00000100:00080000:3.0:1775765194.933140:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765194.933142:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765194.933145:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765194.933153:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066256768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765194.933159:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765194.933162:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765194.933164:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765194.933167:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2026 00000100:00080000:3.0:1775765194.933169:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765194.933171:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765194.933174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765194.933182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066256896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765194.933200:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066256384:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765194.933336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765194.933362:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256640:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765194.933382:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256768:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765194.933411:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256896:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765195.573091:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765196.597021:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765197.621184:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1775765198.005328:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775765198.005377:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775765198.005398:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765198.005444:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765198.005454:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765198.005462:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765198.005471:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775765198.005561:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765194/real 1775765198] req@ffff9d8188740700 x1862023066256896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765198.005603:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066256896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765198.005650:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765198.005660:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765198.005668:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256896:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765198.005683:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765194/real 1775765198] req@ffff9d8188740380 x1862023066256384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765198.005710:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066256384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765198.005725:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765198.005731:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765198.005738:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066256384:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765198.005750:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765194/real 1775765198] req@ffff9d8188740000 x1862023066256512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765198.005772:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066256512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765198.005796:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765198.005802:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765198.005809:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765198.005820:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765194/real 1775765198] req@ffff9d8188742a00 x1862023066256640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765198.005842:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066256640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765198.005856:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765198.005862:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765198.005868:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256640:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765198.005930:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765194/real 1775765198] req@ffff9d8188741c00 x1862023066256768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765198.005966:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066256768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765198.005984:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765198.005990:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765198.005996:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066256768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765198.645085:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765199.669176:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765200.053018:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765200.053031:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765200.053037:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765200.053069:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2031 00000100:00080000:2.0:1775765200.053086:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765200.053100:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765200.053181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1c00 x1862023066257024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765200.053230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765200.053236:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765200.053238:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765200.053242:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2031 00000100:00080000:2.0:1775765200.053245:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765200.053249:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765200.053260:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765200.053271:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0380 x1862023066257152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765200.053279:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765200.053282:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765200.053284:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765200.053288:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2031 00000100:00080000:2.0:1775765200.053290:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765200.053292:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765200.053296:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765200.053304:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066257280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765200.053312:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066257024:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765200.053315:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765200.053321:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765200.053324:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765200.053331:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2031 00000100:00080000:2.0:1775765200.053334:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765200.053336:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765200.053340:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765200.053350:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066257408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765200.053356:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765200.053358:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765200.053359:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765200.053362:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2031 00000100:00080000:2.0:1775765200.053365:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765200.053367:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765200.053370:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765200.053378:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066257536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765200.053422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765200.053487:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765200/real 1775765200] req@ffff9d81984a1c00 x1862023066257024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765200.053509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1c00 x1862023066257024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765200.053522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765200.053527:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765200.053532:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066257024:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765200.053541:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765200.053568:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765200.053583:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765200/real 1775765200] req@ffff9d81984a0380 x1862023066257152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765200.053604:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0380 x1862023066257152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765200.053630:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765200.053634:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765200.053638:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765200.053671:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257280:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765200.053704:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765200.053716:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765200/real 1775765200] req@ffff9d81984a0000 x1862023066257280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765200.053735:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066257280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765200.053744:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765200.053747:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765200.053750:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257280:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765200.053756:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765200.053780:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765200.053792:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765200/real 1775765200] req@ffff9d81984a2a00 x1862023066257408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765200.053804:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066257408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765200.053814:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765200.053818:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765200.053821:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257408:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765200.053827:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257536:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765200.053847:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765200.053858:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765200/real 1775765200] req@ffff9d81984a2d80 x1862023066257536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765200.053869:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066257536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765200.053921:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765200.053924:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765200.053928:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257536:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765200.693028:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765201.717092:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765202.741093:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765203.765074:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765204.789149:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765205.172979:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765205.172983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765205.172991:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765205.172996:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765205.173006:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2036 00000100:00080000:3.0:1775765205.173018:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765205.173025:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765205.173075:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066257664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765205.173100:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765205.173103:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765205.173105:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765205.173108:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2036 00000100:00080000:3.0:1775765205.173110:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765205.173113:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765205.173122:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765205.173130:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066257792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765205.173142:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765205.173144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765205.173145:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765205.173149:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2036 00000100:00080000:3.0:1775765205.173151:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765205.173154:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765205.173158:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765205.173166:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066257920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765205.173173:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765205.173178:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1775765205.173179:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066257664:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765205.173180:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765205.173187:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2036 00000100:00080000:3.0:1775765205.173189:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765205.173192:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765205.173195:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765205.173205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066258048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765205.173211:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765205.173213:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765205.173214:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765205.173217:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2036 00000100:00080000:3.0:1775765205.173221:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765205.173224:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765205.173227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765205.173233:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066258176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765205.173256:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765205.173310:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765205/real 1775765205] req@ffff9d8188742300 x1862023066257664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765205.173326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066257664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765205.173335:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765205.173338:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765205.173342:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066257664:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765205.173347:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765205.173362:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765205.173370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765205/real 1775765205] req@ffff9d8188742680 x1862023066257792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765205.173378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066257792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765205.173384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765205.173386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765205.173388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765205.173408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257920:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765205.173419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765205.173424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765205/real 1775765205] req@ffff9d8188743800 x1862023066257920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765205.173431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066257920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765205.173436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765205.173438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765205.173440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066257920:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765205.173443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765205.173455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765205.173460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765205/real 1775765205] req@ffff9d8188741f80 x1862023066258048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765205.173468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066258048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765205.173474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765205.173475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765205.173478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258048:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765205.173481:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258176:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765205.173492:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765205.173497:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765205/real 1775765205] req@ffff9d8188740a80 x1862023066258176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765205.173504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066258176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765205.173510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765205.173511:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765205.173513:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258176:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765205.813056:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765206.196994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765206.837086:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765207.221017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765207.861129:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765208.244974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765208.885081:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765209.269075:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765209.909191:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765210.292975:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765210.292987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765210.292987:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765210.292993:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765210.293004:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2041 00000100:00080000:3.0:1775765210.293017:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765210.293026:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765210.293085:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066258304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765210.293112:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765210.293115:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765210.293116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765210.293120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2041 00000100:00080000:3.0:1775765210.293123:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765210.293126:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765210.293137:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765210.293148:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066258432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765210.293158:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765210.293159:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066258304:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765210.293169:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765210.293171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765210.293176:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2041 00000100:00080000:3.0:1775765210.293178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765210.293181:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765210.293184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765210.293198:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066258560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765210.293206:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765210.293209:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765210.293210:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765210.293214:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2041 00000100:00080000:3.0:1775765210.293239:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765210.293241:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765210.293246:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765210.293256:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765210.293256:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066258688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765210.293265:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765210.293268:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765210.293270:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765210.293272:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2041 00000100:00080000:3.0:1775765210.293274:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765210.293277:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765210.293282:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765210.293288:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066258816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765210.293332:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765210/real 1775765210] req@ffff9d8188741500 x1862023066258304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765210.293365:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066258304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765210.293378:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765210.293383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765210.293389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066258304:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765210.293418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765210.293443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765210.293457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765210/real 1775765210] req@ffff9d8188741180 x1862023066258432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765210.293471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066258432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765210.293481:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765210.293484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765210.293488:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765210.293495:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258560:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765210.293532:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765210.293544:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765210/real 1775765210] req@ffff9d8188741880 x1862023066258560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765210.293558:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066258560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765210.293574:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765210.293578:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765210.293582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258560:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765210.293589:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765210.293611:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765210.293621:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765210/real 1775765210] req@ffff9d8188743100 x1862023066258688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765210.293634:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066258688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765210.293645:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765210.293649:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765210.293653:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258688:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765210.293659:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258816:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765210.293677:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765210.293688:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765210/real 1775765210] req@ffff9d8188740e00 x1862023066258816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765210.293700:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066258816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765210.293710:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765210.293714:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765210.293718:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066258816:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765210.933079:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765211.316972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765211.957149:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765212.340993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765212.981187:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765213.364968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765214.005029:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765214.388968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765215.029127:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765215.412987:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765215.412998:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765215.413004:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765215.413005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765215.413016:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2046 00000100:00080000:2.0:1775765215.413029:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765215.413037:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765215.413082:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066258944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765215.413113:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765215.413117:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765215.413119:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765215.413123:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2046 00000100:00080000:2.0:1775765215.413127:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765215.413132:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765215.413145:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765215.413154:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066258944:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765215.413162:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066259072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765215.413176:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765215.413187:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765215.413189:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765215.413193:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2046 00000100:00080000:2.0:1775765215.413196:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765215.413200:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765215.413206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765215.413217:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066259200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765215.413236:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765215.413239:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765215.413240:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765215.413243:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2046 00000100:00080000:2.0:1775765215.413246:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765215.413251:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765215.413253:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775765215.413258:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765215.413269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066259328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765215.413276:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765215.413279:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765215.413279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765215.413282:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2046 00000100:00080000:2.0:1775765215.413284:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765215.413286:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765215.413290:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765215.413309:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066259456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765215.413319:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765215/real 1775765215] req@ffff9d81984a0700 x1862023066258944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765215.413336:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066258944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765215.413346:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765215.413350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765215.413353:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066258944:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765215.413375:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765215.413404:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765215.413412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765215/real 1775765215] req@ffff9d81984a3b80 x1862023066259072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765215.413420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066259072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765215.413427:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765215.413429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765215.413431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765215.413435:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259200:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765215.413446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765215.413451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765215/real 1775765215] req@ffff9d81984a3480 x1862023066259200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765215.413458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066259200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765215.413477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765215.413479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765215.413481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259200:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765215.413486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765215.413502:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765215.413516:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765215/real 1775765215] req@ffff9d81984a2d80 x1862023066259328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765215.413524:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066259328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765215.413531:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765215.413533:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765215.413536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259328:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765215.413540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259456:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765215.413552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765215.413558:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765215/real 1775765215] req@ffff9d81984a2a00 x1862023066259456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765215.413565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066259456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765215.413572:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765215.413575:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765215.413578:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259456:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765216.053078:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765216.436983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765217.077237:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765217.461041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765218.101065:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765218.484966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765219.125128:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765219.509053:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765220.149001:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765220.532964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765220.533274:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066259584:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765220.533359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765220.533449:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765220/real 1775765220] req@ffff9d81901ad180 x1862023066259584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765220.533488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066259584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765220.533498:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765220.533501:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765220.533505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066259584:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765220.533511:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765220.533531:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765220.533541:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765220/real 1775765220] req@ffff9d81901af800 x1862023066259712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765220.533550:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066259712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765220.533556:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765220.533559:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765220.533561:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765220.533589:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259840:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765220.533613:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765220.533625:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765220/real 1775765220] req@ffff9d81901afb80 x1862023066259840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765220.533641:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066259840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765220.533651:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765220.533655:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765220.533658:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259840:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765220.533664:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765220.533681:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765220.533689:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765220/real 1775765220] req@ffff9d81901aed80 x1862023066259968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765220.533698:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066259968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765220.533705:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765220.533707:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765220.533709:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066259968:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765220.533713:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260096:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765220.533725:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765220.533731:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765220/real 1775765220] req@ffff9d81901af100 x1862023066260096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765220.533740:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066260096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765220.533746:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765220.533748:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765220.533750:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260096:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765221.173135:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765221.557011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765222.197143:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765222.581006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765223.221138:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765223.605008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765224.245066:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765224.629003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765225.269204:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765225.652990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765225.652996:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765225.653009:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765225.653015:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765225.653027:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2057 00000100:00080000:3.0:1775765225.653040:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765225.653048:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765225.653090:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066260224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765225.653130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765225.653135:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765225.653137:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765225.653143:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2057 00000100:00080000:3.0:1775765225.653150:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765225.653154:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765225.653166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765225.653179:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066260224:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765225.653190:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066260352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765225.653205:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765225.653210:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765225.653215:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765225.653221:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2057 00000100:00080000:3.0:1775765225.653226:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765225.653230:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765225.653236:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765225.653248:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066260480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765225.653261:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765225.653265:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765225.653267:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765225.653271:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2057 00000100:00080000:3.0:1775765225.653291:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765225.653298:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765225.653312:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765225.653320:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765225.653355:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066260608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765225.653387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765225/real 1775765225] req@ffff9d8188743480 x1862023066260224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765225.653405:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765225.653409:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775765225.653413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066260224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765225.653419:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765225.653435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765225.653436:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2057 00000100:00080000:1.0:1775765225.653449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775765225.653453:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765225.653459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066260224:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765225.653466:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765225.653478:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765225.653497:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066260736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765225.653499:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765225.653564:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765225.653578:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765225/real 1775765225] req@ffff9d8188742d80 x1862023066260352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765225.653588:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066260352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765225.653595:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765225.653598:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765225.653601:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765225.653605:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260480:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765225.653633:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765225.653641:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765225/real 1775765225] req@ffff9d8188743b80 x1862023066260480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765225.653650:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066260480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765225.653655:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765225.653658:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765225.653660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260480:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765225.653664:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765225.653685:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765225.653692:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765225/real 1775765225] req@ffff9d8188740e00 x1862023066260608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765225.653699:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066260608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765225.653706:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765225.653709:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765225.653711:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260608:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765225.653723:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260736:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765225.653734:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765225.653740:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765225/real 1775765225] req@ffff9d8188743100 x1862023066260736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765225.653748:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066260736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765225.653754:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765225.653757:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765225.653759:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260736:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765226.293207:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765226.676999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765227.317141:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765227.701006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765228.341114:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765228.724957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765229.365010:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765229.748954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765230.389134:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765230.773041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765230.773385:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066260864:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775765230.773505:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765230.773597:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765230/real 1775765230] req@ffff9d81901ae680 x1862023066260864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765230.773635:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066260864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765230.773657:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765230.773666:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765230.773674:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066260864:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765230.773713:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765230.773760:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765230.773786:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765230/real 1775765230] req@ffff9d81901af100 x1862023066260992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765230.773827:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066260992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765230.773845:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765230.773852:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765230.773859:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066260992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765230.773873:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261120:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765230.773959:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765230.773983:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765230/real 1775765230] req@ffff9d81901aed80 x1862023066261120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765230.774009:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066261120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765230.774027:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765230.774034:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765230.774040:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261120:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765230.774052:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765230.774092:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765230.774115:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765230/real 1775765230] req@ffff9d81901afb80 x1862023066261248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765230.774140:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066261248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765230.774169:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765230.774176:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765230.774182:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261248:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765230.774194:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261376:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765230.774235:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765230.774258:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765230/real 1775765230] req@ffff9d81901af800 x1862023066261376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765230.774284:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066261376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765230.774302:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765230.774309:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765230.774316:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261376:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765231.413005:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765231.796948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765232.437014:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765232.820941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765233.461130:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765233.845041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765234.485075:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765234.868942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765235.509075:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765235.892990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765235.893022:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765235.893031:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765235.893036:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765235.893047:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2067 00000100:00080000:2.0:1775765235.893058:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765235.893065:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765235.893109:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066261504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765235.893154:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765235.893158:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765235.893160:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765235.893163:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2067 00000100:00080000:2.0:1775765235.893166:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765235.893170:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765235.893181:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765235.893207:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0380 x1862023066261632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765235.893209:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066261504:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765235.893237:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765235.893239:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765235.893240:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765235.893244:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2067 00000100:00080000:2.0:1775765235.893247:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765235.893252:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765235.893256:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765235.893269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1c00 x1862023066261760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765235.893289:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765235.893292:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765235.893294:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765235.893297:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2067 00000100:00080000:2.0:1775765235.893306:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765235.893310:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765235.893317:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775765235.893317:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765235.893332:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066261888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765235.893341:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765235.893344:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765235.893345:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765235.893348:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2067 00000100:00080000:2.0:1775765235.893351:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765235.893353:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765235.893356:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765235.893363:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066262016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765235.893394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765235/real 1775765235] req@ffff9d81984a0000 x1862023066261504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765235.893414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066261504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765235.893427:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765235.893432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765235.893436:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066261504:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765235.893464:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765235.893486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765235.893497:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765235/real 1775765235] req@ffff9d81984a0380 x1862023066261632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765235.893510:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0380 x1862023066261632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765235.893520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765235.893523:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765235.893526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765235.893532:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261760:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765235.893548:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765235.893556:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765235/real 1775765235] req@ffff9d81984a1c00 x1862023066261760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765235.893568:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1c00 x1862023066261760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765235.893583:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765235.893586:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765235.893589:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261760:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765235.893596:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765235.893615:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765235.893625:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765235/real 1775765235] req@ffff9d81984a2300 x1862023066261888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765235.893637:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066261888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765235.893646:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765235.893650:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765235.893654:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066261888:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765235.893660:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262016:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765235.893680:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765235.893690:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765235/real 1775765235] req@ffff9d81984a1f80 x1862023066262016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765235.893702:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066262016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765235.893711:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765235.893714:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765235.893718:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262016:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765236.533175:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765236.917028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765237.557076:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765237.940998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765238.581117:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765238.965015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765239.605094:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765239.988999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765240.629185:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765241.012956:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765241.012965:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765241.012969:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765241.012980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765241.012980:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2072 00000100:00080000:3.0:1775765241.012992:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765241.013000:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765241.013033:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066262144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765241.013055:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765241.013058:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765241.013059:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765241.013061:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2072 00000100:00080000:3.0:1775765241.013064:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765241.013067:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765241.013075:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765241.013086:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066262272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765241.013093:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066262144:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765241.013104:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765241.013108:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765241.013109:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765241.013112:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2072 00000100:00080000:3.0:1775765241.013114:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765241.013116:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765241.013120:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765241.013141:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066262400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765241.013148:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765241.013150:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765241.013151:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765241.013153:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2072 00000100:00080000:3.0:1775765241.013174:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765241.013177:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765241.013186:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765241.013189:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765241.013211:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066262528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765241.013224:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765241.013244:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765241.013247:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765241.013253:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2072 00000100:00080000:3.0:1775765241.013259:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765241.013264:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765241.013269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765241/real 1775765241] req@ffff9d8188741880 x1862023066262144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775765241.013269:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765241.013300:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066262144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765241.013301:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066262656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765241.013371:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765241.013376:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765241.013380:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066262144:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765241.013407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765241.013427:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765241.013436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765241/real 1775765241] req@ffff9d8188741180 x1862023066262272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765241.013444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066262272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765241.013451:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765241.013453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765241.013456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765241.013459:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262400:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765241.013470:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765241.013476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765241/real 1775765241] req@ffff9d8188741500 x1862023066262400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765241.013483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066262400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765241.013488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765241.013490:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765241.013492:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262400:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765241.013496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765241.013507:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765241.013513:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765241/real 1775765241] req@ffff9d8188740a80 x1862023066262528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765241.013520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066262528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765241.013526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765241.013528:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765241.013530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262528:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765241.013534:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262656:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765241.013545:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765241.013550:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765241/real 1775765241] req@ffff9d8188741f80 x1862023066262656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765241.013557:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066262656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765241.013562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765241.013564:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765241.013566:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262656:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765241.653029:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765242.036984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765242.677117:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765243.060968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765243.701269:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765244.084994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765244.725047:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765245.108955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765245.749126:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765246.133049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765246.133099:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765246.133114:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765246.133137:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765246.133150:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2077 00000100:00080000:3.0:1775765246.133163:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765246.133172:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765246.133266:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066262784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765246.133304:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765246.133309:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765246.133311:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765246.133316:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2077 00000100:00080000:3.0:1775765246.133321:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765246.133328:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765246.133343:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765246.133363:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066262912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765246.133379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066262784:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765246.133381:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765246.133388:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765246.133392:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765246.133410:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2077 00000100:00080000:3.0:1775765246.133414:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765246.133417:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765246.133424:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765246.133438:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066263040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765246.133446:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765246.133450:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765246.133451:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765246.133455:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2077 00000100:00080000:3.0:1775765246.133480:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765246.133483:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765246.133489:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765246.133499:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765246.133509:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066263168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765246.133521:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765246.133526:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765246.133529:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765246.133533:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2077 00000100:00080000:3.0:1775765246.133537:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765246.133541:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765246.133548:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765246.133560:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066263296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765246.133574:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765246/real 1775765246] req@ffff9d8188743800 x1862023066262784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765246.133609:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066262784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765246.133622:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765246.133628:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765246.133634:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066262784:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765246.133671:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765246.133722:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765246.133747:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765246/real 1775765246] req@ffff9d8188742680 x1862023066262912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765246.133766:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066262912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765246.133776:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765246.133780:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765246.133784:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066262912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765246.133792:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263040:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765246.133817:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765246.133829:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765246/real 1775765246] req@ffff9d8188742300 x1862023066263040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765246.133843:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066263040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765246.133859:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765246.133863:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765246.133867:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263040:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765246.133873:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765246.133930:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765246.133942:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765246/real 1775765246] req@ffff9d8188741c00 x1862023066263168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765246.133955:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066263168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765246.133966:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765246.133970:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765246.133974:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263168:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765246.133981:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263296:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765246.133999:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765246.134008:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765246/real 1775765246] req@ffff9d8188742a00 x1862023066263296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765246.134021:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066263296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765246.134030:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765246.134033:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765246.134037:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263296:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765246.773151:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765247.156928:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765247.797149:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765248.181044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765248.821103:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765249.205056:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765249.845072:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765250.229021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765250.869012:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765251.252995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765251.253270:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066263424:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775765251.253373:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765251.253452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765251/real 1775765251] req@ffff9d81901ad180 x1862023066263424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765251.253478:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066263424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765251.253493:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765251.253499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765251.253505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066263424:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765251.253515:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765251.253542:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765251.253575:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765251/real 1775765251] req@ffff9d81901af800 x1862023066263552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765251.253591:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066263552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765251.253602:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765251.253606:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765251.253611:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765251.253640:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263680:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765251.253663:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765251.253675:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765251/real 1775765251] req@ffff9d81901afb80 x1862023066263680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765251.253689:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066263680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765251.253698:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765251.253702:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765251.253706:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263680:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765251.253713:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765251.253732:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765251.253742:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765251/real 1775765251] req@ffff9d81901aed80 x1862023066263808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765251.253756:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066263808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765251.253766:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765251.253770:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765251.253774:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263808:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765251.253781:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263936:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765251.253800:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765251.253810:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765251/real 1775765251] req@ffff9d81901af100 x1862023066263936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765251.253823:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066263936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765251.253833:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765251.253837:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765251.253840:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066263936:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765251.893148:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765252.276965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765252.917161:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765253.300992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765253.941145:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765254.325010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765254.965094:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765255.349036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765255.989038:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765256.372959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765256.373153:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066264064:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775765256.373214:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765256.373269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765256/real 1775765256] req@ffff9d81901ae680 x1862023066264064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765256.373317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066264064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765256.373326:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765256.373330:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765256.373333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066264064:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765256.373351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765256.373368:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765256.373377:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765256/real 1775765256] req@ffff9d81901af100 x1862023066264192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765256.373385:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066264192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765256.373391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765256.373393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765256.373395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765256.373399:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264320:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765256.373409:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765256.373415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765256/real 1775765256] req@ffff9d81901aed80 x1862023066264320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765256.373423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066264320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765256.373433:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765256.373435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765256.373438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264320:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765256.373441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765256.373452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765256.373457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765256/real 1775765256] req@ffff9d81901afb80 x1862023066264448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765256.373465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066264448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765256.373471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765256.373473:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765256.373475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264448:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765256.373479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264576:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765256.373489:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765256.373495:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765256/real 1775765256] req@ffff9d81901af800 x1862023066264576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765256.373503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066264576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765256.373508:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765256.373510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765256.373513:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264576:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765257.013012:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765257.397030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765258.037170:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765258.421009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765259.061145:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765259.444953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765260.085174:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765260.469016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765261.109186:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765261.493032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765261.493265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066264704:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765261.493413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765261.493458:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264960:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765261.493500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265088:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765261.493581:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265216:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765262.133117:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765262.517153:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765263.157195:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765263.541072:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765264.181101:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765264.564953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775765264.565205:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775765264.565240:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775765264.565256:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765264.565288:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765264.565300:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765264.565313:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775765264.565326:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765261/real 1775765264] req@ffff9d81901af800 x1862023066264832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1775765264.565327:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775765264.565360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066264832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765264.565375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765264.565381:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765264.565386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765264.565396:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765261/real 1775765264] req@ffff9d81901afb80 x1862023066264960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765264.565411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066264960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765264.565421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765264.565425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765264.565443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066264960:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765264.565452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765261/real 1775765264] req@ffff9d81901ad180 x1862023066264704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765264.565467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066264704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765264.565477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765264.565481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765264.565486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066264704:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765264.565510:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765261/real 1775765264] req@ffff9d81901af100 x1862023066265216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765264.565527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066265216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765264.565538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765264.565542:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765264.565546:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265216:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765264.565554:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765261/real 1775765264] req@ffff9d81901aed80 x1862023066265088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765264.565568:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066265088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765264.565577:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765264.565580:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765264.565582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765265.205215:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765265.588997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765266.229057:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765266.612932:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765266.612976:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765266.612982:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765266.612985:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765266.613005:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2098 00000100:00080000:3.0:1775765266.613014:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765266.613019:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765266.613048:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066265344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765266.613068:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765266.613071:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765266.613072:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765266.613075:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2098 00000100:00080000:3.0:1775765266.613078:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765266.613081:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765266.613089:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765266.613097:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066265472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765266.613105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765266.613107:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765266.613109:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765266.613111:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2098 00000100:00080000:3.0:1775765266.613114:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765266.613117:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066265344:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765266.613118:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765266.613126:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765266.613134:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066265600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765266.613141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765266.613143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765266.613145:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765266.613148:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2098 00000100:00080000:3.0:1775765266.613150:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765266.613153:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765266.613156:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765266.613163:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066265728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765266.613175:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765266.613176:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765266.613180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765266.613182:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765266.613185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2098 00000100:00080000:3.0:1775765266.613190:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765266.613193:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765266.613207:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765266.613218:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765266/real 1775765266] req@ffff9d8188740000 x1862023066265344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765266.613226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066265856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765266.613243:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066265344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765266.613255:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765266.613262:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765266.613267:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066265344:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765266.613277:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765266.613301:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765266.613314:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765266/real 1775765266] req@ffff9d8188740380 x1862023066265472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765266.613326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066265472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765266.613335:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765266.613339:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765266.613342:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765266.613362:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265600:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765266.613398:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765266.613413:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765266/real 1775765266] req@ffff9d8188740700 x1862023066265600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765266.613428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066265600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765266.613443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765266.613447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765266.613449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265600:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765266.613455:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765266.613476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765266.613488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765266/real 1775765266] req@ffff9d8188742a00 x1862023066265728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765266.613501:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066265728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765266.613511:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765266.613515:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765266.613518:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265728:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765266.613524:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265856:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765266.613543:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765266.613554:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765266/real 1775765266] req@ffff9d8188741c00 x1862023066265856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765266.613568:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066265856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765266.613576:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765266.613580:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765266.613583:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066265856:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765267.253174:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765267.637037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765268.277159:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765268.660948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765269.301170:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765269.685009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765270.325211:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765270.708994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765271.349118:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765271.733027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765271.733108:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765271.733122:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765271.733127:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765271.733140:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2103 00000100:00080000:3.0:1775765271.733154:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765271.733163:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765271.733211:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066265984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765271.733253:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765271.733258:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765271.733260:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765271.733265:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2103 00000100:00080000:3.0:1775765271.733278:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765271.733282:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765271.733296:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765271.733310:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066266112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765271.733323:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765271.733328:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765271.733331:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765271.733337:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2103 00000100:00080000:3.0:1775765271.733344:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765271.733356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066265984:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765271.733360:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765271.733377:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765271.733393:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066266240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765271.733407:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765271.733412:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765271.733415:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765271.733420:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2103 00000100:00080000:3.0:1775765271.733425:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765271.733429:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765271.733436:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765271.733449:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066266368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765271.733461:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765271.733465:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765271.733468:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765271.733473:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2103 00000100:00080000:3.0:1775765271.733493:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765271.733498:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765271.733505:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:1.0:1775765271.733515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765271.733524:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066266496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765271.733658:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765271/real 1775765271] req@ffff9d8188742300 x1862023066265984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765271.733686:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066265984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765271.733702:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765271.733708:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765271.733714:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066265984:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765271.733726:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765271.733760:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765271.733774:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765271/real 1775765271] req@ffff9d8188742680 x1862023066266112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765271.733789:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066266112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765271.733817:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765271.733821:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765271.733825:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765271.733862:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266240:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765271.733886:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765271.733937:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765271/real 1775765271] req@ffff9d8188743800 x1862023066266240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765271.733996:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066266240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765271.734008:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765271.734012:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765271.734016:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266240:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765271.734024:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765271.734053:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765271.734066:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765271/real 1775765271] req@ffff9d8188741f80 x1862023066266368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765271.734079:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066266368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765271.734090:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765271.734094:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765271.734097:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266368:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765271.734104:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266496:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765271.734123:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765271.734132:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765271/real 1775765271] req@ffff9d8188740a80 x1862023066266496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765271.734145:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066266496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765271.734156:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765271.734160:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765271.734164:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266496:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765272.373036:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765272.757140:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765273.397073:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765273.781059:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765274.421151:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765274.804992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765275.445173:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765275.828972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765276.469119:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765276.852965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765276.852965:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765276.852975:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765276.852979:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765276.852988:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2108 00000100:00080000:3.0:1775765276.852997:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765276.853004:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765276.853036:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066266624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765276.853062:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765276.853065:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765276.853066:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765276.853069:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2108 00000100:00080000:3.0:1775765276.853072:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765276.853074:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765276.853082:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765276.853092:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066266752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765276.853106:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765276.853109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765276.853110:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765276.853114:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2108 00000100:00080000:3.0:1775765276.853116:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765276.853119:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765276.853122:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765276.853131:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066266624:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765276.853138:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066266880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765276.853157:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765276.853162:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765276.853163:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765276.853170:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2108 00000100:00080000:3.0:1775765276.853175:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765276.853180:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765276.853188:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765276.853203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066267008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765276.853217:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765276.853221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1775765276.853225:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765276.853227:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765276.853235:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2108 00000100:00080000:3.0:1775765276.853244:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765276.853250:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765276.853257:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765276.853275:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066267136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765276.853293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765276/real 1775765276] req@ffff9d8188741500 x1862023066266624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765276.853312:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066266624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765276.853322:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765276.853326:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765276.853329:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066266624:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765276.853336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765276.853355:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765276.853363:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765276/real 1775765276] req@ffff9d8188741180 x1862023066266752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765276.853372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066266752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765276.853379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765276.853381:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765276.853384:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765276.853406:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266880:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765276.853417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765276.853423:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765276/real 1775765276] req@ffff9d8188741880 x1862023066266880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765276.853431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066266880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765276.853437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765276.853439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765276.853441:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066266880:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765276.853445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765276.853457:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765276.853463:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765276/real 1775765276] req@ffff9d8188743100 x1862023066267008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765276.853471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066267008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765276.853477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765276.853479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765276.853481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267008:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765276.853486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267136:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765276.853497:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765276.853502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765276/real 1775765276] req@ffff9d8188740e00 x1862023066267136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765276.853509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066267136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765276.853515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765276.853517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765276.853519:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267136:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765277.493146:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765277.877090:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765278.517116:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765278.900970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765279.541041:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765279.924989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765280.565124:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765280.948980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765281.589078:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765281.972945:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765281.972954:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775765281.972957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765281.972958:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765281.972969:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2113 00000100:00080000:3.0:1775765281.972978:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765281.972985:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765281.973015:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066267264/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765281.973039:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765281.973042:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765281.973044:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765281.973047:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2113 00000100:00080000:3.0:1775765281.973049:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765281.973052:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765281.973059:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765281.973072:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066267392/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765281.973085:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765281.973090:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066267264:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765281.973096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765281.973099:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765281.973104:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2113 00000100:00080000:3.0:1775765281.973110:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765281.973117:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765281.973123:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765281.973138:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066267520/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765281.973148:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765281.973151:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765281.973152:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765281.973155:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2113 00000100:00080000:3.0:1775765281.973165:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765281.973168:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765281.973174:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765281.973177:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765281.973190:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066267648/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765281.973197:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765281.973200:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765281.973201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765281.973204:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2113 00000100:00080000:3.0:1775765281.973207:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765281.973210:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765281.973215:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765281.973220:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765281/real 1775765281] req@ffff9d8188743b80 x1862023066267264/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765281.973225:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066267776/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765281.973268:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066267264/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765281.973284:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765281.973288:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765281.973291:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066267264:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765281.973311:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765281.973332:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765281.973341:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765281/real 1775765281] req@ffff9d8188742d80 x1862023066267392/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765281.973350:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066267392/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765281.973356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765281.973358:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765281.973361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765281.973364:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267520:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765281.973376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765281.973382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765281/real 1775765281] req@ffff9d8188743480 x1862023066267520/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765281.973389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066267520/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765281.973399:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765281.973402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765281.973404:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267520:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765281.973425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765281.973440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765281.973446:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765281/real 1775765281] req@ffff9d8188740e00 x1862023066267648/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765281.973455:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066267648/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765281.973462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765281.973465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765281.973467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267648:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765281.973471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267776:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765281.973483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765281.973489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765281/real 1775765281] req@ffff9d8188743100 x1862023066267776/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765281.973498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066267776/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765281.973504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765281.973506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765281.973509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066267776:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765282.613032:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765282.996966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765283.637017:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765284.020984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765284.661005:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765285.044977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765285.685129:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765286.069043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765286.709043:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765287.093010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765287.093054:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765287.093066:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765287.093072:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765287.093084:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2118 00000100:00080000:2.0:1775765287.093096:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765287.093118:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765287.093179:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066267904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765287.093241:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765287.093246:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765287.093249:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765287.093255:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2118 00000100:00080000:2.0:1775765287.093263:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765287.093270:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765287.093293:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765287.093308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066267904:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765287.093314:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066268032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765287.093346:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765287.093350:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765287.093352:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765287.093358:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2118 00000100:00080000:2.0:1775765287.093363:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765287.093368:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765287.093373:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765287.093398:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066268160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765287.093408:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765287.093412:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765287.093414:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765287.093419:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2118 00000100:00080000:2.0:1775765287.093452:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765287.093456:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765287.093462:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765287.093472:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765287.093489:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066268288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765287.093523:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765287.093528:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765287.093530:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765287.093535:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2118 00000100:00080000:2.0:1775765287.093540:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765287.093555:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765287.093561:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765287.093574:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066268416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765287.093634:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765287/real 1775765287] req@ffff9d81984a1180 x1862023066267904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765287.093658:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066267904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765287.093670:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765287.093674:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765287.093678:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066267904:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765287.093722:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765287.093749:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765287.093759:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765287/real 1775765287] req@ffff9d81984a1500 x1862023066268032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765287.093772:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066268032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765287.093784:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765287.093823:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765287.093828:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765287.093836:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268160:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765287.093863:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765287.093900:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765287/real 1775765287] req@ffff9d81984a2680 x1862023066268160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765287.093911:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066268160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765287.093917:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765287.093920:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765287.093923:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268160:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765287.093928:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765287.093948:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765287.093956:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765287/real 1775765287] req@ffff9d81984a1880 x1862023066268288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765287.093965:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066268288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765287.093973:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765287.093975:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765287.093978:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268288:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765287.093983:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268416:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765287.093995:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765287.094001:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765287/real 1775765287] req@ffff9d81984a3800 x1862023066268416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765287.094010:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066268416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765287.094016:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765287.094018:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765287.094020:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268416:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765287.733052:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765288.116977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765288.757012:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765289.140966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765289.781140:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765290.164946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765290.805022:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765291.188955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765291.829021:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765292.212956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765292.213024:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765292.213037:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765292.213042:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765292.213054:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2123 00000100:00080000:3.0:1775765292.213068:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765292.213077:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765292.213124:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066268544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765292.213175:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765292.213180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765292.213183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765292.213188:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2123 00000100:00080000:3.0:1775765292.213192:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765292.213198:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765292.213222:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765292.213238:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066268672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765292.213252:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066268544:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765292.213276:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765292.213282:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765292.213287:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765292.213299:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2123 00000100:00080000:3.0:1775765292.213309:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765292.213318:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765292.213330:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765292.213360:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066268800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765292.213362:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765292.213385:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765292.213392:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765292.213396:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765292.213408:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2123 00000100:00080000:3.0:1775765292.213418:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765292.213428:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765292.213431:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765292/real 1775765292] req@ffff9d8188741880 x1862023066268544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775765292.213444:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765292.213476:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066268928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765292.213485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066268544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765292.213517:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765292.213527:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765292.213529:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765292.213531:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765292.213544:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775765292.213544:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2123 00000100:00080000:3.0:1775765292.213564:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765292.213575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066268544:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765292.213575:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765292.213598:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765292.213619:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066269056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765292.213631:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765292.213704:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765292.213726:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765292/real 1775765292] req@ffff9d8188741180 x1862023066268672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765292.213741:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066268672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765292.213749:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765292.213772:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765292.213776:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765292.213807:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268800:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765292.213832:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765292.213845:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765292/real 1775765292] req@ffff9d8188741500 x1862023066268800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765292.213860:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066268800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765292.213872:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765292.213901:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765292.213904:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268800:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765292.213910:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765292.213937:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765292.213949:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765292/real 1775765292] req@ffff9d8188740a80 x1862023066268928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765292.213959:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066268928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765292.213973:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765292.213979:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765292.213983:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066268928:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765292.214019:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269056:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765292.214040:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765292.214049:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765292/real 1775765292] req@ffff9d8188741f80 x1862023066269056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765292.214057:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066269056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765292.214063:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765292.214066:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765292.214069:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269056:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765292.853196:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765293.236975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765293.877055:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765294.260944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765294.901089:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765295.285005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765295.925020:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765296.309001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765296.949038:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765297.332943:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765297.332951:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765297.332956:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765297.332957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765297.332963:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2128 00000100:00080000:3.0:1775765297.332972:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765297.332978:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765297.333012:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066269184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765297.333035:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765297.333038:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765297.333040:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765297.333050:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2128 00000100:00080000:3.0:1775765297.333053:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765297.333056:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765297.333064:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765297.333068:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066269184:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765297.333081:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066269312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765297.333090:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765297.333093:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765297.333094:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765297.333098:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2128 00000100:00080000:3.0:1775765297.333112:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765297.333114:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765297.333119:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775765297.333121:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765297.333131:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066269440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765297.333138:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765297.333140:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765297.333142:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765297.333144:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2128 00000100:00080000:3.0:1775765297.333147:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765297.333149:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765297.333155:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765297.333164:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765297/real 1775765297] req@ffff9d8188743800 x1862023066269184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765297.333165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066269568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765297.333195:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765297.333197:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765297.333199:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765297.333203:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066269184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765297.333203:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2128 00000100:00080000:3.0:1775765297.333208:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765297.333214:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765297.333221:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:1.0:1775765297.333222:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765297.333227:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765297.333230:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066269696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765297.333234:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066269184:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765297.333260:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765297.333277:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765297.333285:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765297/real 1775765297] req@ffff9d8188742680 x1862023066269312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765297.333292:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066269312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765297.333299:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765297.333301:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765297.333303:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765297.333307:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269440:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765297.333324:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765297.333335:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765297/real 1775765297] req@ffff9d8188742300 x1862023066269440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765297.333346:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066269440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765297.333354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765297.333357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765297.333360:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269440:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765297.333366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765297.333384:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765297.333392:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765297/real 1775765297] req@ffff9d8188741c00 x1862023066269568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765297.333399:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066269568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765297.333406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765297.333408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765297.333410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269568:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765297.333414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269696:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765297.333424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765297.333429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765297/real 1775765297] req@ffff9d8188742a00 x1862023066269696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765297.333436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066269696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765297.333441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765297.333443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765297.333444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269696:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765297.973034:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765298.356962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765298.997159:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765299.380955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765300.021138:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765300.404941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765301.045205:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765301.428962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765302.069047:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765302.452959:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765302.452964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765302.452968:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765302.452973:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765302.452981:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2133 00000100:00080000:3.0:1775765302.452990:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765302.452996:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765302.453027:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066269824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765302.453050:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765302.453053:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765302.453054:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765302.453057:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2133 00000100:00080000:3.0:1775765302.453059:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765302.453062:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765302.453071:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765302.453080:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066269824:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765302.453082:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066269952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765302.453096:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765302.453100:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765302.453102:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765302.453106:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2133 00000100:00080000:3.0:1775765302.453110:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765302.453114:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765302.453120:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775765302.453136:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765302.453142:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066270080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765302.453155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765302.453157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765302.453159:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765302.453163:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2133 00000100:00080000:3.0:1775765302.453166:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765302.453169:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765302.453174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765302.453175:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765302/real 1775765302] req@ffff9d8188740700 x1862023066269824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765302.453188:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066270208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765302.453196:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765302.453200:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775765302.453201:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066269824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765302.453202:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765302.453209:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2133 00000100:00080000:3.0:1775765302.453214:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765302.453217:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765302.453218:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:1.0:1775765302.453225:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:3.0:1775765302.453226:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765302.453229:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066269824:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765302.453242:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066270336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765302.453250:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765302.453294:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765302.453311:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765302/real 1775765302] req@ffff9d8188740380 x1862023066269952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765302.453324:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066269952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765302.453349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765302.453352:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765302.453355:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066269952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765302.453361:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270080:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765302.453381:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765302.453392:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765302/real 1775765302] req@ffff9d8188740000 x1862023066270080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765302.453405:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066270080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765302.453420:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765302.453423:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765302.453425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270080:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765302.453429:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765302.453444:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765302.453451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765302/real 1775765302] req@ffff9d8188742a00 x1862023066270208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765302.453459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066270208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765302.453465:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765302.453467:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765302.453468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270208:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765302.453478:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270336:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765302.453488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765302.453493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765302/real 1775765302] req@ffff9d8188741c00 x1862023066270336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765302.453500:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066270336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765302.453505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765302.453507:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765302.453509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270336:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765303.093165:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765303.476967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765304.117104:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765304.500983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765305.141146:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765305.524953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765306.165083:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765306.548949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765307.189019:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765307.572927:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765307.572935:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765307.572941:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765307.572952:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2139 00000100:00100000:1.0:1775765307.572955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765307.572964:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765307.572972:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765307.573012:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066270464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765307.573046:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765307.573049:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765307.573050:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765307.573053:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2139 00000100:00080000:3.0:1775765307.573057:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765307.573060:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765307.573068:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765307.573081:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066270592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765307.573082:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066270464:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765307.573096:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765307.573098:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765307.573099:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765307.573101:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2139 00000100:00080000:3.0:1775765307.573103:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765307.573105:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765307.573114:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765307.573127:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066270720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765307.573133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765307.573136:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765307.573138:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765307.573141:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2139 00000100:00080000:3.0:1775765307.573143:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765307.573145:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765307.573149:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765307.573150:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765307.573160:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066270848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765307.573167:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765307.573170:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765307.573171:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765307.573173:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2139 00000100:00080000:3.0:1775765307.573177:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765307.573181:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765307.573186:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765307.573191:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765307/real 1775765307] req@ffff9d8188742300 x1862023066270464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765307.573199:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066270976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765307.573208:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066270464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765307.573219:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765307.573222:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765307.573225:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066270464:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765307.573242:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765307.573258:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765307.573265:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765307/real 1775765307] req@ffff9d8188742680 x1862023066270592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765307.573273:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066270592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765307.573279:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765307.573281:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765307.573284:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765307.573287:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270720:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765307.573297:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765307.573302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765307/real 1775765307] req@ffff9d8188743800 x1862023066270720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765307.573309:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066270720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765307.573314:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765307.573315:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765307.573317:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270720:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765307.573321:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765307.573330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765307.573336:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765307/real 1775765307] req@ffff9d8188741f80 x1862023066270848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765307.573343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066270848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765307.573348:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765307.573350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765307.573352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270848:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765307.573356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270976:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765307.573365:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765307.573370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765307/real 1775765307] req@ffff9d8188740a80 x1862023066270976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765307.573377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066270976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765307.573405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765307.573407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765307.573410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066270976:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765308.213078:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765308.597001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765309.237064:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765309.620963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765310.261019:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765310.645004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765311.285109:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765311.668962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765312.308989:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765312.692983:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765312.692988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765312.692995:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765312.693001:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765312.693013:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2144 00000100:00080000:3.0:1775765312.693025:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765312.693033:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765312.693075:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066271104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765312.693113:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765312.693118:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765312.693120:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765312.693124:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2144 00000100:00080000:3.0:1775765312.693128:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765312.693133:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765312.693147:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765312.693169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066271232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765312.693189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765312.693190:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066271104:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765312.693205:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765312.693208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765312.693216:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2144 00000100:00080000:3.0:1775765312.693224:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765312.693230:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765312.693252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775765312.693270:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765312.693274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066271360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765312.693288:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765312.693293:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765312.693296:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765312.693300:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2144 00000100:00080000:3.0:1775765312.693321:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765312.693324:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765312.693329:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765312.693343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765312/real 1775765312] req@ffff9d8188741500 x1862023066271104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765312.693343:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066271488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765312.693364:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765312.693371:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765312.693373:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765312.693396:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2144 00000100:00080000:3.0:1775765312.693405:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765312.693414:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765312.693420:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765312.693434:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066271616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765312.693452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066271104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765312.693466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765312.693472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765312.693477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066271104:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765312.693503:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765312.693531:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765312.693545:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765312/real 1775765312] req@ffff9d8188741180 x1862023066271232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765312.693558:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066271232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765312.693611:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765312.693632:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765312.693634:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765312.693639:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271360:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765312.693654:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765312.693662:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765312/real 1775765312] req@ffff9d8188741880 x1862023066271360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765312.693670:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066271360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765312.693680:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765312.693682:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765312.693684:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271360:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765312.693687:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765312.693697:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765312.693703:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765312/real 1775765312] req@ffff9d8188743100 x1862023066271488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765312.693710:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066271488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765312.693716:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765312.693717:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765312.693719:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271488:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765312.693722:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271616:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765312.693732:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765312.693738:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765312/real 1775765312] req@ffff9d8188740e00 x1862023066271616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765312.693745:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066271616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765312.693750:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765312.693752:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765312.693754:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271616:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765313.333011:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765313.716959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765314.357122:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765314.740982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765315.381041:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765315.764965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765316.405167:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765316.789044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765317.429112:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765317.812936:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765317.812943:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765317.812948:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765317.812955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765317.812956:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2149 00000100:00080000:3.0:1775765317.812967:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765317.812974:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765317.813006:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066271744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765317.813027:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765317.813032:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765317.813033:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765317.813036:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2149 00000100:00080000:3.0:1775765317.813038:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765317.813041:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765317.813049:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765317.813057:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066271872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765317.813063:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066271744:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765317.813071:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765317.813074:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765317.813076:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765317.813080:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2149 00000100:00080000:3.0:1775765317.813083:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765317.813086:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765317.813089:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765317.813098:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066272000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765317.813105:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765317.813107:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765317.813108:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765317.813111:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2149 00000100:00080000:3.0:1775765317.813115:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765317.813117:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765317.813122:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765317.813126:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765317.813135:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066272128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765317.813145:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765317.813149:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765317.813151:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765317.813156:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2149 00000100:00080000:3.0:1775765317.813160:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765317.813166:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765317/real 1775765317] req@ffff9d8188743480 x1862023066271744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765317.813167:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765317.813182:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765317.813187:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066271744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765317.813196:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775765317.813200:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066272256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765317.813203:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765317.813210:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066271744:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765317.813235:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765317.813251:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765317.813259:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765317/real 1775765317] req@ffff9d8188742d80 x1862023066271872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765317.813267:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066271872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765317.813272:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765317.813275:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765317.813278:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066271872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765317.813284:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272000:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765317.813303:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765317.813314:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765317/real 1775765317] req@ffff9d8188743b80 x1862023066272000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765317.813326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066272000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765317.813334:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765317.813338:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765317.813341:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272000:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765317.813346:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765317.813366:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765317.813378:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765317/real 1775765317] req@ffff9d8188740e00 x1862023066272128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765317.813390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066272128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765317.813399:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765317.813402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765317.813407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272128:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765317.813412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272256:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765317.813427:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765317.813434:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765317/real 1775765317] req@ffff9d8188743100 x1862023066272256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765317.813442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066272256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765317.813447:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765317.813449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765317.813451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272256:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765318.453120:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765318.837025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765319.477117:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765319.860970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765320.501032:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765320.885029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765321.525053:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765321.908969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765322.549102:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765322.933013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765322.933101:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765322.933111:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765322.933117:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765322.933133:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2154 00000100:00080000:2.0:1775765322.933150:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765322.933163:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765322.933220:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066272384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765322.933267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765322.933276:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765322.933279:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765322.933284:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2154 00000100:00080000:2.0:1775765322.933318:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765322.933328:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066272384:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765322.933338:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765322.933358:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765322.933397:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066272512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765322.933422:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765322.933428:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765322.933432:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775765322.933444:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765322.933451:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2154 00000100:00080000:2.0:1775765322.933463:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765322.933505:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765322.933516:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765322.933544:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066272640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765322.933565:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765322.933570:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765322/real 1775765322] req@ffff9d81984a3100 x1862023066272384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765322.933576:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765322.933582:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765322.933599:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2154 00000100:00080000:2.0:1775765322.933611:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765322.933626:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765322.933641:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765322.933656:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066272384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765322.933671:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066272768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765322.933690:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765322.933705:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775765322.933709:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765322.933715:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066272384:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765322.933726:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765322.933732:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765322.933744:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2154 00000100:00080000:2.0:1775765322.933752:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765322.933762:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272512:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775765322.933762:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765322.933778:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765322.933803:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066272896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775765322.933828:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765322.933917:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765322/real 1775765322] req@ffff9d81984a0e00 x1862023066272512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765322.933943:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066272512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765322.933954:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765322.933959:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765322.933963:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765322.933981:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272640:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765322.934011:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765322.934026:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765322/real 1775765322] req@ffff9d81984a0a80 x1862023066272640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765322.934040:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066272640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765322.934058:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765322.934063:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765322.934067:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272640:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765322.934074:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765322.934093:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765322.934103:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765322/real 1775765322] req@ffff9d81984a3800 x1862023066272768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765322.934116:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066272768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765322.934128:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765322.934132:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765322.934135:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272768:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765322.934156:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272896:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765322.934175:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765322.934185:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765322/real 1775765322] req@ffff9d81984a1880 x1862023066272896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765322.934200:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066272896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765322.934209:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765322.934213:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765322.934217:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066272896:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765323.573133:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765323.957026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765324.597107:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765324.980965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765325.621094:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765326.005037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765326.645008:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765327.029007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765327.669115:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765328.052982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765328.053030:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765328.053037:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765328.053041:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765328.053049:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2159 00000100:00080000:3.0:1775765328.053059:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765328.053065:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765328.053103:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066273024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765328.053131:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765328.053135:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765328.053136:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765328.053139:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2159 00000100:00080000:3.0:1775765328.053143:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765328.053147:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765328.053157:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765328.053169:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066273024:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765328.053170:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066273152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765328.053187:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765328.053190:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765328.053192:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765328.053196:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2159 00000100:00080000:3.0:1775765328.053199:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765328.053201:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765328.053204:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765328.053213:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066273280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765328.053220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765328.053223:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765328.053225:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765328.053228:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2159 00000100:00080000:3.0:1775765328.053239:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765328.053242:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765328.053247:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765328.053255:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066273408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765328.053269:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765328.053272:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765328.053274:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765328.053278:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2159 00000100:00080000:3.0:1775765328.053281:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765328.053285:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765328.053294:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765328.053303:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765328.053309:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066273536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765328.053341:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273280:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765328.053358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273408:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765328.053378:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273536:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765328.693029:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765329.717068:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765330.741148:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1775765331.125261:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775765331.125310:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775765331.125330:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765331.125377:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765331.125417:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765331.125440:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765331.125463:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775765331.125477:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765328/real 1775765331] req@ffff9d8188741180 x1862023066273152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765331.125559:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066273152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765331.125583:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765331.125593:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765331.125603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765331.125618:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765328/real 1775765331] req@ffff9d8188741500 x1862023066273280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765331.125642:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066273280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765331.125658:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765331.125664:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765331.125671:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273280:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765331.125684:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765328/real 1775765331] req@ffff9d8188740a80 x1862023066273408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765331.125707:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066273408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765331.125726:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765331.125732:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765331.125738:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273408:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765331.125749:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765328/real 1775765331] req@ffff9d8188741880 x1862023066273024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765331.125771:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066273024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765331.125785:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765331.125792:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765331.125798:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066273024:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765331.125840:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765328/real 1775765331] req@ffff9d8188741f80 x1862023066273536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765331.125866:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066273536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765331.125924:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765331.125931:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765331.125938:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273536:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765331.765196:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765332.789176:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765333.172975:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765333.173001:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765333.173008:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765333.173023:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2164 00000100:00080000:3.0:1775765333.173039:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765333.173050:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765333.173106:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066273664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765333.173142:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765333.173147:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765333.173149:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765333.173160:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2164 00000100:00080000:3.0:1775765333.173164:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765333.173168:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765333.173183:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765333.173198:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066273792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765333.173210:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765333.173214:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765333.173216:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765333.173222:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2164 00000100:00080000:3.0:1775765333.173227:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765333.173231:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765333.173237:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765333.173259:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066273664:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765333.173265:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066273920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765333.173286:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765333.173292:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765333.173295:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765333.173302:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2164 00000100:00080000:3.0:1775765333.173311:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765333.173319:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765333.173329:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765333.173349:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066274048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765333.173363:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775765333.173364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765333.173392:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765333.173397:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765333.173407:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2164 00000100:00080000:3.0:1775765333.173414:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765333.173435:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765333.173446:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765333.173479:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066274176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765333.173489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765333/real 1775765333] req@ffff9d8188743800 x1862023066273664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765333.173555:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066273664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765333.173574:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765333.173582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765333.173588:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066273664:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765333.173625:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765333.173666:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765333.173687:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765333/real 1775765333] req@ffff9d8188742680 x1862023066273792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765333.173723:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066273792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765333.173738:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765333.173744:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765333.173749:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765333.173789:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273920:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765333.173826:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765333.173845:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765333/real 1775765333] req@ffff9d8188742300 x1862023066273920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765333.173865:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066273920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765333.173924:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765333.173931:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765333.173937:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066273920:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765333.173948:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765333.173986:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765333.174005:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765333/real 1775765333] req@ffff9d8188741c00 x1862023066274048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765333.174024:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066274048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765333.174050:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765333.174056:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765333.174061:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274048:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765333.174071:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274176:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765333.174105:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765333.174122:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765333/real 1775765333] req@ffff9d8188742a00 x1862023066274176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765333.174139:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066274176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765333.174150:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765333.174156:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765333.174161:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274176:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765333.813136:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765334.837170:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765335.861222:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765336.885049:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765337.909130:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765338.293053:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765338.293067:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765338.293074:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765338.293086:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2169 00000100:00080000:3.0:1775765338.293096:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765338.293136:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765338.293199:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066274304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765338.293254:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765338.293258:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765338.293260:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765338.293265:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2169 00000100:00080000:3.0:1775765338.293271:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765338.293277:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765338.293290:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765338.293291:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066274304:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765338.293315:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066274432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765338.293359:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765338.293363:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765338.293381:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765338.293386:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2169 00000100:00080000:3.0:1775765338.293397:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765338.293402:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765338.293407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765338.293410:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765338.293428:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066274560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765338.293438:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765338.293441:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765338.293443:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765338.293448:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2169 00000100:00080000:3.0:1775765338.293453:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765338.293456:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765338.293462:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765338/real 1775765338] req@ffff9d8188740000 x1862023066274304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775765338.293463:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765338.293479:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066274688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765338.293487:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066274304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765338.293493:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765338.293504:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765338.293508:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765338.293514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765338.293514:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2169 00000100:00080000:3.0:1775765338.293523:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765338.293525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775765338.293530:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765338.293535:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765338.293536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066274304:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765338.293548:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066274816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765338.293580:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765338.293605:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765338.293633:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765338/real 1775765338] req@ffff9d8188740380 x1862023066274432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765338.293650:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066274432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765338.293661:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765338.293665:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765338.293669:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765338.293677:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274560:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765338.293699:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765338.293710:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765338/real 1775765338] req@ffff9d8188740700 x1862023066274560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765338.293724:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066274560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765338.293733:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765338.293737:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765338.293741:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274560:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765338.293748:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765338.293767:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765338.293777:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765338/real 1775765338] req@ffff9d8188742a00 x1862023066274688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765338.293790:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066274688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765338.293801:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765338.293805:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765338.293808:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274688:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765338.293815:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274816:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765338.293832:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765338.293842:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765338/real 1775765338] req@ffff9d8188741c00 x1862023066274816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765338.293856:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066274816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765338.293865:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765338.293869:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765338.293872:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066274816:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765338.933048:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765339.316967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765339.957137:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765340.341046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765340.981157:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765341.364983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765342.005091:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765342.388969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765343.029074:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765343.413018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775765343.413150:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066274944:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775765343.413243:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765343.413302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765343/real 1775765343] req@ffff9d81901ae680 x1862023066274944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765343.413321:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066274944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765343.413331:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765343.413334:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765343.413336:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066274944:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765343.413353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765343.413370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765343.413378:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765343/real 1775765343] req@ffff9d81901aed80 x1862023066275072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765343.413386:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066275072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765343.413391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765343.413393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765343.413395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765343.413407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275200:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765343.413420:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765343.413426:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765343/real 1775765343] req@ffff9d81901af100 x1862023066275200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765343.413433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066275200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765343.413438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765343.413440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765343.413442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275200:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765343.413445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765343.413455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765343.413460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765343/real 1775765343] req@ffff9d81901ad180 x1862023066275328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765343.413467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066275328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765343.413473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765343.413475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765343.413476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275328:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765343.413479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275456:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765343.413488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765343.413493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765343/real 1775765343] req@ffff9d81901afb80 x1862023066275456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765343.413505:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066275456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765343.413510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765343.413529:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765343.413533:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275456:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765344.053198:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765344.436943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765345.077069:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765345.460971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765346.101127:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765346.484958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765347.125002:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765347.508952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765348.149153:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765348.533015:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765348.533046:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765348.533053:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765348.533068:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2180 00000100:00080000:3.0:1775765348.533089:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775765348.533092:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765348.533100:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765348.533178:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066275584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765348.533263:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765348.533269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765348.533272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765348.533279:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2180 00000100:00080000:3.0:1775765348.533284:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765348.533290:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765348.533305:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765348.533321:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066275712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765348.533347:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765348.533354:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765348.533357:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765348.533361:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066275584:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765348.533366:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2180 00000100:00080000:3.0:1775765348.533373:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765348.533378:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765348.533383:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765348.533412:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066275840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765348.533423:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765348.533427:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765348.533429:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765348.533432:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2180 00000100:00080000:3.0:1775765348.533435:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765348.533439:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765348.533445:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765348.533456:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066275968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765348.533465:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765348.533468:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765348.533471:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765348.533475:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2180 00000100:00080000:3.0:1775765348.533478:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765348.533481:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765348.533488:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:1.0:1775765348.533493:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765348.533503:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066276096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765348.533867:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765348/real 1775765348] req@ffff9d8188742300 x1862023066275584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765348.533950:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066275584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765348.533967:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765348.533975:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765348.533984:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066275584:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765348.534019:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765348.534102:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765348.534130:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765348/real 1775765348] req@ffff9d8188742680 x1862023066275712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765348.534176:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066275712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765348.534194:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765348.534218:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765348.534224:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765348.534263:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275840:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765348.534296:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765348.534312:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765348/real 1775765348] req@ffff9d8188743800 x1862023066275840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765348.534327:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066275840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765348.534336:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765348.534340:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765348.534344:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275840:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765348.534351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765348.534393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765348.534405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765348/real 1775765348] req@ffff9d8188741f80 x1862023066275968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765348.534419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066275968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765348.534439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765348.534443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765348.534447:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066275968:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765348.534457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276096:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765348.534508:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765348.534526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765348/real 1775765348] req@ffff9d8188741880 x1862023066276096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765348.534548:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066276096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765348.534564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765348.534570:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765348.534577:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276096:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765349.173115:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765350.197143:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765351.221093:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765352.245147:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765353.269076:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765353.652960:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765353.652970:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765353.652974:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765353.652982:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2185 00000100:00080000:3.0:1775765353.652994:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765353.653001:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765353.653039:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066276224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765353.653064:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765353.653067:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765353.653069:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765353.653072:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2185 00000100:00080000:3.0:1775765353.653074:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765353.653077:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765353.653085:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765353.653095:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066276352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765353.653102:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765353.653105:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765353.653106:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765353.653109:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2185 00000100:00080000:3.0:1775765353.653113:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765353.653115:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765353.653119:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765353.653122:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066276224:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765353.653129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066276480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765353.653137:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765353.653139:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765353.653140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765353.653142:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2185 00000100:00080000:3.0:1775765353.653145:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765353.653147:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765353.653151:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765353.653162:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066276608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765353.653172:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765353.653175:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765353.653177:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765353.653181:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2185 00000100:00080000:3.0:1775765353.653202:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765353.653206:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775765353.653211:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765353.653215:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765353.653231:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066276736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765353.653265:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765353/real 1775765353] req@ffff9d8188740a80 x1862023066276224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765353.653295:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066276224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765353.653306:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765353.653311:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765353.653315:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066276224:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765353.653322:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765353.653346:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765353.653358:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765353/real 1775765353] req@ffff9d8188741500 x1862023066276352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765353.653372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066276352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765353.653380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765353.653392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765353.653396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765353.653419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276480:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765353.653440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765353.653450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765353/real 1775765353] req@ffff9d8188741180 x1862023066276480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765353.653462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066276480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765353.653470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765353.653474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765353.653477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276480:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765353.653483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765353.653501:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765353.653512:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765353/real 1775765353] req@ffff9d8188743100 x1862023066276608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765353.653523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066276608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765353.653532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765353.653536:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765353.653539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276608:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765353.653546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276736:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765353.653565:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765353.653575:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765353/real 1775765353] req@ffff9d8188740e00 x1862023066276736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765353.653584:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066276736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765353.653590:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765353.653592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765353.653594:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276736:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765354.293033:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765355.317230:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765356.341076:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765357.365056:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765358.389069:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765358.772956:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765358.772965:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765358.772970:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765358.772978:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2190 00000100:00080000:3.0:1775765358.772988:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765358.772995:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765358.773029:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066276864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765358.773055:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765358.773058:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765358.773060:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765358.773062:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2190 00000100:00080000:3.0:1775765358.773066:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765358.773069:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765358.773078:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765358.773088:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066276992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765358.773106:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066276864:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765358.773111:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765358.773120:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765358.773123:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765358.773129:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2190 00000100:00080000:3.0:1775765358.773135:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765358.773158:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765358.773167:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765358.773185:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066277120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765358.773199:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775765358.773202:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765358.773204:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765358.773207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765358.773213:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2190 00000100:00080000:3.0:1775765358.773217:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765358.773222:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765358.773227:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765358.773240:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066277248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765358.773248:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765358.773253:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765358/real 1775765358] req@ffff9d8188743b80 x1862023066276864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765358.773253:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765358.773258:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765358.773266:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2190 00000100:00080000:3.0:1775765358.773272:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765358.773279:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765358.773284:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765358.773296:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066277376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765358.773297:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066276864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765358.773324:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765358.773329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765358.773333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066276864:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765358.773354:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765358.773376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765358.773387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765358/real 1775765358] req@ffff9d8188742d80 x1862023066276992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765358.773397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066276992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765358.773404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765358.773407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765358.773410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066276992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765358.773415:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277120:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765358.773441:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765358.773448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765358/real 1775765358] req@ffff9d8188743480 x1862023066277120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765358.773457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066277120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765358.773463:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765358.773465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765358.773468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277120:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765358.773472:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765358.773485:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765358.773502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765358/real 1775765358] req@ffff9d8188740e00 x1862023066277248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765358.773511:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066277248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765358.773518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765358.773521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765358.773523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277248:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765358.773527:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277376:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765358.773538:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765358.773545:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765358/real 1775765358] req@ffff9d8188743100 x1862023066277376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765358.773559:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066277376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765358.773566:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765358.773568:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765358.773570:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277376:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765359.413029:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765360.437042:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765361.461073:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765362.485099:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765363.509051:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765363.892974:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765363.892980:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765363.892983:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765363.892991:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2195 00000100:00080000:2.0:1775765363.893000:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765363.893006:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765363.893038:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066277504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765363.893064:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765363.893067:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765363.893068:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765363.893071:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2195 00000100:00080000:2.0:1775765363.893073:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765363.893075:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765363.893082:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765363.893109:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066277632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765363.893117:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765363.893119:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765363.893121:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765363.893125:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2195 00000100:00080000:2.0:1775765363.893128:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765363.893130:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066277504:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765363.893133:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765363.893142:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765363.893152:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066277760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765363.893159:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765363.893161:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765363.893163:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765363.893165:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2195 00000100:00080000:2.0:1775765363.893179:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765363.893182:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765363.893185:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765363.893189:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765363.893197:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066277888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765363.893205:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765363.893207:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765363.893208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765363.893211:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2195 00000100:00080000:2.0:1775765363.893213:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765363.893215:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765363.893218:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765363.893227:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066278016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765363.893233:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765363/real 1775765363] req@ffff9d81984a2680 x1862023066277504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765363.893249:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066277504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765363.893257:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765363.893261:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765363.893264:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066277504:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765363.893268:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765363.893283:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765363.893290:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765363/real 1775765363] req@ffff9d81984a1500 x1862023066277632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765363.893298:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066277632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765363.893304:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765363.893306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765363.893308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765363.893324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277760:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765363.893334:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765363.893343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765363/real 1775765363] req@ffff9d81984a1180 x1862023066277760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765363.893354:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066277760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765363.893362:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765363.893365:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765363.893369:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277760:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765363.893375:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765363.893395:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765363.893405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765363/real 1775765363] req@ffff9d81984a1f80 x1862023066277888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765363.893413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066277888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765363.893424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765363.893426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765363.893428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066277888:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765363.893432:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278016:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765363.893445:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765363.893450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765363/real 1775765363] req@ffff9d81984a2300 x1862023066278016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765363.893457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066278016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765363.893462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765363.893464:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765363.893465:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278016:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765364.533200:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765365.557148:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765366.581118:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765367.605131:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765368.629056:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765369.013000:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765369.013006:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765369.013010:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765369.013017:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2200 00000100:00080000:3.0:1775765369.013026:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765369.013033:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765369.013084:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066278144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765369.013113:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765369.013116:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765369.013118:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765369.013121:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2200 00000100:00080000:3.0:1775765369.013124:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765369.013127:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765369.013136:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765369.013146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066278272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765369.013154:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066278144:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765369.013156:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765369.013163:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765369.013166:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765369.013169:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2200 00000100:00080000:3.0:1775765369.013171:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765369.013173:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765369.013176:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765369.013183:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066278400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765369.013194:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765369.013196:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765369.013198:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765369.013200:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2200 00000100:00080000:3.0:1775765369.013215:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765369.013217:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765369.013221:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775765369.013225:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765369.013230:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066278528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765369.013237:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765369.013240:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765369.013241:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765369.013244:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2200 00000100:00080000:3.0:1775765369.013246:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765369.013248:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765369.013252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765369.013260:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066278656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765369.013267:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765369/real 1775765369] req@ffff9d8188741180 x1862023066278144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765369.013284:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066278144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765369.013294:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765369.013298:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765369.013302:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066278144:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765369.013326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765369.013350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765369.013371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765369/real 1775765369] req@ffff9d8188741500 x1862023066278272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765369.013381:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066278272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765369.013387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765369.013389:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765369.013391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765369.013395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278400:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775765369.013409:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765369.013415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765369/real 1775765369] req@ffff9d8188740a80 x1862023066278400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765369.013423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066278400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765369.013428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765369.013430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765369.013432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278400:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765369.013436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765369.013446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765369.013451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765369/real 1775765369] req@ffff9d8188741880 x1862023066278528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765369.013458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066278528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765369.013464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765369.013466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765369.013467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278528:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775765369.013471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278656:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775765369.013480:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765369.013485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765369/real 1775765369] req@ffff9d8188741f80 x1862023066278656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765369.013492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066278656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765369.013497:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765369.013498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765369.013500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066278656:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765369.653070:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765370.677120:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765371.701115:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765372.725084:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765373.749126:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765374.132926:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765374.132938:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765374.132945:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765374.132958:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2205 00000100:00080000:3.0:1775765374.132973:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765374.132983:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765374.133053:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066278784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765374.133091:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765374.133100:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765374.133103:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765374.133111:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2205 00000100:00080000:3.0:1775765374.133117:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765374.133122:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765374.133134:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765374.133151:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066278912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765374.133161:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765374.133170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765374.133178:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765374.133208:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2205 00000100:00080000:3.0:1775765374.133212:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765374.133216:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765374.133221:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765374.133235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066279040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765374.133243:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765374.133247:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765374.133249:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765374.133252:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2205 00000100:00080000:3.0:1775765374.133266:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765374.133268:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765374.133274:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765374.133287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066279168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765374.133302:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765374.133306:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765374.133309:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765374.133315:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2205 00000100:00080000:3.0:1775765374.133322:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765374.133328:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765374.133338:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765374.133363:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066279296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765374.773141:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765375.157049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765375.797088:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765376.181072:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765376.821172:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765377.205047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765377.845163:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765378.228988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765378.869072:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765379.253088:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765379.253123:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765379.253136:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765379.253144:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765379.253159:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2210 00000100:00080000:2.0:1775765379.253175:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765379.253187:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765379.253241:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1c00 x1862023066279424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765379.253285:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765379.253293:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765379.253297:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765379.253306:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2210 00000100:00080000:2.0:1775765379.253316:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765379.253325:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765379.253342:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765379.253374:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0380 x1862023066279552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765379.253405:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765379.253410:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765379.253413:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765379.253421:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2210 00000100:00080000:2.0:1775765379.253430:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765379.253438:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765379.253449:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765379.253474:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066279680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765379.253494:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765379.253501:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765379.253505:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765379.253536:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2210 00000100:00080000:2.0:1775765379.253547:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765379.253555:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765379.253564:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765379.253585:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066279808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765379.253598:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765379.253603:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765379.253607:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765379.253616:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2210 00000100:00080000:2.0:1775765379.253623:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765379.253630:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765379.253648:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765379.253666:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066279936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765379.893082:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765380.277077:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765380.917121:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765381.301023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765381.941214:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765382.325080:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765382.965132:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765383.349033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765383.989116:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765384.372991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765384.372998:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765384.373005:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765384.373009:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765384.373019:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2215 00000100:00080000:2.0:1775765384.373030:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765384.373038:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765384.373076:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066280064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765384.373107:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765384.373111:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765384.373113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765384.373118:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2215 00000100:00080000:2.0:1775765384.373122:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765384.373125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765384.373135:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765384.373147:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066280192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765384.373157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765384.373161:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765384.373166:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765384.373174:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2215 00000100:00080000:2.0:1775765384.373185:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765384.373189:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765384.373191:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765384.373205:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066280320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765384.373220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765384.373224:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765384.373226:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765384.373231:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2215 00000100:00080000:2.0:1775765384.373252:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765384.373257:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765384.373263:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765384.373285:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066280448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765384.373294:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765384.373296:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765384.373298:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765384.373301:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2215 00000100:00080000:2.0:1775765384.373304:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765384.373307:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765384.373312:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765384.373324:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066280576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765385.013109:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765385.396999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765386.037155:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765386.420951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765387.061037:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765387.444923:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765388.085073:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765388.468964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765389.109052:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765389.493021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765389.493290:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066280704:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765389.493372:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765389.493438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765389/real 1775765389] req@ffff9d81901af800 x1862023066280704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765389.493476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066280704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765389.493506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765389.493512:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765389.493517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066280704:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765389.493526:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066280832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765389.493552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765389.493564:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765389/real 1775765389] req@ffff9d81901afb80 x1862023066280832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765389.493575:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066280832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765389.493582:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765389.493585:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765389.493588:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066280832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765389.493611:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066280960:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765389.493626:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765389.493634:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765389/real 1775765389] req@ffff9d81901ad180 x1862023066280960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765389.493644:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066280960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765389.493650:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765389.493652:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765389.493655:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066280960:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765389.493659:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765389.493675:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765389.493687:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765389/real 1775765389] req@ffff9d81901af100 x1862023066281088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765389.493703:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066281088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765389.493715:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765389.493719:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765389.493723:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281088:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765389.493732:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281216:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765389.493758:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765389.493774:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765389/real 1775765389] req@ffff9d81901aed80 x1862023066281216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765389.493791:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066281216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765389.493802:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765389.493806:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765389.493809:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281216:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765390.133119:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765390.517075:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765391.157098:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765391.540971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765392.181080:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765392.565047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765393.205033:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765393.588933:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765394.229063:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765394.612952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765394.613011:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765394.613019:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765394.613024:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765394.613034:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2226 00000100:00080000:1.0:1775765394.613046:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765394.613053:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765394.613098:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066281344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765394.613125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765394.613128:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765394.613129:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765394.613132:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2226 00000100:00080000:1.0:1775765394.613134:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765394.613137:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765394.613146:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765394.613157:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066281472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765394.613165:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765394.613168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765394.613170:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765394.613175:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066281344:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765394.613176:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2226 00000100:00080000:1.0:1775765394.613181:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765394.613184:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765394.613188:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765394.613196:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066281600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765394.613203:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765394.613206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765394.613207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765394.613210:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2226 00000100:00080000:1.0:1775765394.613212:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765394.613216:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765394.613219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765394.613226:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066281728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765394.613232:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765394.613234:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765394.613235:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765394.613238:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2226 00000100:00080000:1.0:1775765394.613248:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765394.613250:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765394.613253:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765394.613261:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066281856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765394.613318:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765394.613363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281600:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765394.613392:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281728:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765394.613422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281856:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765395.253035:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765395.637028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765396.277075:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765396.660981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765397.301068:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765397.684985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775765397.685098:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775765397.685128:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775765397.685142:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765397.685176:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765397.685187:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765397.685204:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765397.685212:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765394/real 1775765397] req@ffff9d8190551880 x1862023066281472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1775765397.685220:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765397.685249:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066281472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765397.685260:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765397.685265:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765397.685270:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765397.685278:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765394/real 1775765397] req@ffff9d8190550e00 x1862023066281600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765397.685291:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066281600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765397.685299:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765397.685303:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765397.685307:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281600:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765397.685313:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765394/real 1775765397] req@ffff9d8190553100 x1862023066281728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765397.685325:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066281728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765397.685335:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765397.685338:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765397.685341:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281728:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765397.685348:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765394/real 1775765397] req@ffff9d8190553480 x1862023066281856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765397.685358:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066281856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765397.685367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765397.685370:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765397.685374:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066281856:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765397.685380:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765394/real 1775765397] req@ffff9d8190551500 x1862023066281344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765397.685391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066281344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765397.685398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765397.685402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765397.685406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066281344:192.168.202.105@tcp:250:kworker.0 00010000:00010000:1.0:1775765398.325055:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765398.709007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765399.349092:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775765399.732960:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765399.732969:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765399.732973:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765399.732981:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2231 00000100:00080000:1.0:1775765399.733003:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765399.733003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765399.733011:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765399.733041:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553800 x1862023066281984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765399.733061:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765399.733064:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765399.733065:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765399.733074:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2231 00000100:00080000:1.0:1775765399.733077:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765399.733084:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765399.733090:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066281984:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775765399.733092:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765399.733120:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066282112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765399.733133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765399.733136:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765399.733138:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765399.733142:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2231 00000100:00080000:1.0:1775765399.733154:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765399.733159:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765399.733165:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775765399.733168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765399.733184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066282240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765399.733194:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765399.733198:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765399.733200:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765399.733206:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2231 00000100:00080000:1.0:1775765399.733211:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765399.733215:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765399.733220:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765399/real 1775765399] req@ffff9d8190553800 x1862023066281984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775765399.733222:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765399.733241:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066282368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765399.733246:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553800 x1862023066281984/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765399.733267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765399.733269:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775765399.733274:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765399.733277:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765399.733277:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765399.733285:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066281984:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765399.733288:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2231 00000100:00080000:1.0:1775765399.733295:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765399.733299:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765399.733303:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765399.733314:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775765399.733315:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066282496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765399.733347:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765399.733361:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765399/real 1775765399] req@ffff9d8190551500 x1862023066282112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765399.733375:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066282112/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765399.733382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765399.733386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765399.733390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765399.733396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282240:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765399.733411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765399.733420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765399/real 1775765399] req@ffff9d8190553480 x1862023066282240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765399.733433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066282240/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765399.733441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765399.733445:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765399.733448:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282240:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765399.733454:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765399.733469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765399.733485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765399/real 1775765399] req@ffff9d8190553100 x1862023066282368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765399.733497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066282368/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765399.733514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765399.733517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765399.733521:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282368:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765399.733535:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282496:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765399.733551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765399.733622:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765399/real 1775765399] req@ffff9d8190550e00 x1862023066282496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765399.733636:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066282496/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765399.733645:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765399.733648:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765399.733652:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282496:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765400.373018:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765400.757041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765401.397118:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765401.781017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765402.421009:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765402.805007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765403.445105:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765403.828968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765404.469045:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765404.853003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765404.853011:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765404.853021:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765404.853027:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765404.853039:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2236 00000100:00080000:1.0:1775765404.853054:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765404.853081:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765404.853129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066282624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765404.853166:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765404.853171:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765404.853173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765404.853178:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2236 00000100:00080000:1.0:1775765404.853182:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765404.853186:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765404.853198:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765404.853216:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066282752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765404.853231:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765404.853240:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066282624:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765404.853241:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765404.853245:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765404.853258:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2236 00000100:00080000:1.0:1775765404.853262:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765404.853266:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765404.853271:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765404.853288:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066282880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765404.853299:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765404.853304:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765404.853307:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765404.853313:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2236 00000100:00080000:1.0:1775765404.853330:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765404.853335:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765404.853342:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765404.853347:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765404.853363:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066283008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765404.853377:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765404.853380:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765404.853383:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765404.853389:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2236 00000100:00080000:1.0:1775765404.853394:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765404.853399:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765404.853407:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765404.853416:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765404/real 1775765404] req@ffff9d8190551880 x1862023066282624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765404.853424:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066283136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765404.853476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066282624/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765404.853492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765404.853498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765404.853503:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066282624:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765404.853537:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765404.853567:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765404.853582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765404/real 1775765404] req@ffff9d8190550e00 x1862023066282752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765404.853597:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066282752/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765404.853607:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765404.853612:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765404.853616:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765404.853622:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282880:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765404.853640:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765404.853651:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765404/real 1775765404] req@ffff9d8190553100 x1862023066282880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765404.853664:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066282880/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765404.853686:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765404.853690:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765404.853695:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066282880:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765404.853729:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066283008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765404.853756:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765404.853770:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765404/real 1775765404] req@ffff9d8190553480 x1862023066283008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765404.853784:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066283008/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765404.853795:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765404.853799:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765404.853803:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066283008:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765404.853809:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066283136:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765404.853828:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765404.853837:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765404/real 1775765404] req@ffff9d8190551500 x1862023066283136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765404.853850:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066283136/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765404.853859:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765404.853863:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765404.853866:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066283136:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765405.493041:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765405.876980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765406.517042:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765406.900959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765407.541004:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765407.925020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765408.565033:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765408.948951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765409.589010:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765409.972944:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765409.972952:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765409.972956:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765409.972965:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2241 00000100:00080000:3.0:1775765409.972975:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765409.972980:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765409.973012:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066283264/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765409.973040:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765409.973043:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765409.973045:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765409.973055:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2241 00000100:00080000:3.0:1775765409.973058:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765409.973063:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765409.973072:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765409.973085:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066283392/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765409.973096:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765409.973099:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765409.973100:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765409.973104:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2241 00000100:00080000:3.0:1775765409.973107:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765409.973110:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765409.973115:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765409.973125:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066283520/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765409.973133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765409.973136:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765409.973138:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765409.973141:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2241 00000100:00080000:3.0:1775765409.973144:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765409.973147:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765409.973152:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765409.973161:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066283648/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765409.973169:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765409.973172:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765409.973174:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765409.973177:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2241 00000100:00080000:3.0:1775765409.973181:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765409.973184:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765409.973188:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765409.973196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066283776/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765409.973224:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765410.613009:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765410.996937:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765411.637069:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765412.020933:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765412.661013:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765413.044949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765413.685088:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765414.068968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765414.709026:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765415.092933:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765415.092939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765415.092945:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765415.092951:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765415.092962:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2246 00000100:00080000:3.0:1775765415.092973:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765415.092980:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765415.093021:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066283904/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765415.093045:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765415.093047:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765415.093049:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765415.093052:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2246 00000100:00080000:3.0:1775765415.093054:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765415.093057:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765415.093067:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765415.093076:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066284032/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765415.093084:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765415.093087:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765415.093088:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765415.093094:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2246 00000100:00080000:3.0:1775765415.093097:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765415.093100:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765415.093107:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765415.093114:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066284160/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765415.093120:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765415.093123:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765415.093125:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765415.093128:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2246 00000100:00080000:3.0:1775765415.093130:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765415.093132:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765415.093135:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765415.093144:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066284288/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765415.093150:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765415.093153:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765415.093154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765415.093157:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2246 00000100:00080000:3.0:1775765415.093171:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765415.093174:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765415.093177:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765415.093190:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066284416/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765415.733117:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765416.116974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765416.757120:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765417.141041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765417.781074:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765418.165009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765418.805058:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765419.188983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765419.829028:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765420.212961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765420.212975:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765420.212984:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765420.212989:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765420.212998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2251 00000100:00080000:3.0:1775765420.213007:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765420.213013:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765420.213048:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066284544/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765420.213073:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765420.213076:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765420.213077:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765420.213080:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2251 00000100:00080000:3.0:1775765420.213082:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765420.213084:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765420.213092:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765420.213099:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066284672/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765420.213107:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765420.213109:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765420.213110:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765420.213113:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2251 00000100:00080000:3.0:1775765420.213115:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765420.213118:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765420.213121:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765420.213129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066284800/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765420.213146:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765420.213149:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765420.213150:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765420.213152:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2251 00000100:00080000:3.0:1775765420.213154:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765420.213156:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765420.213159:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765420.213167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066284928/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765420.213173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765420.213175:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765420.213176:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765420.213179:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2251 00000100:00080000:3.0:1775765420.213181:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765420.213183:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765420.213186:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765420.213197:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066285056/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765420.853064:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765421.877053:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765422.901055:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765423.925041:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765424.949098:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765425.332918:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765425.332927:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765425.332931:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765425.332939:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2256 00000100:00080000:3.0:1775765425.332948:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765425.332955:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765425.332989:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066285184/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765425.333006:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765425.333009:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765425.333011:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765425.333015:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2256 00000100:00080000:3.0:1775765425.333019:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765425.333021:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765425.333029:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765425.333040:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066285312/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765425.333064:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765425.333066:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765425.333067:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765425.333071:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2256 00000100:00080000:3.0:1775765425.333073:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765425.333076:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765425.333079:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765425.333087:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066285440/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765425.333094:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765425.333098:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765425.333100:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765425.333104:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2256 00000100:00080000:3.0:1775765425.333129:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765425.333135:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765425.333142:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765425.333162:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066285568/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765425.333171:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765425.333174:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765425.333175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765425.333179:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2256 00000100:00080000:3.0:1775765425.333183:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765425.333186:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765425.333190:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765425.333205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066285696/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765425.973024:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765426.997042:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765428.021002:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765429.045115:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765430.069068:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765430.452992:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765430.453005:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765430.453012:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765430.453025:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2261 00000100:00080000:3.0:1775765430.453043:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765430.453063:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765430.453113:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066285824/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765430.453150:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765430.453155:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765430.453157:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765430.453162:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2261 00000100:00080000:3.0:1775765430.453167:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765430.453172:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765430.453184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765430.453203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066285952/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765430.453234:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765430.453239:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765430.453241:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765430.453248:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2261 00000100:00080000:3.0:1775765430.453255:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765430.453262:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765430.453273:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765430.453308:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066286080/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765430.453329:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765430.453354:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765430.453370:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765430.453380:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2261 00000100:00080000:3.0:1775765430.453389:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765430.453399:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765430.453410:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765430.453440:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066286208/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765430.453509:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765430.453516:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765430.453521:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765430.453535:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2261 00000100:00080000:3.0:1775765430.453546:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765430.453561:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765430.453570:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765430.453592:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066286336/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765431.093172:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765432.117137:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765433.141119:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765434.165113:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765435.189060:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765435.572970:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765435.572991:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765435.572994:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765435.573003:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2267 00000100:00080000:3.0:1775765435.573013:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765435.573030:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765435.573079:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066286464/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765435.573106:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765435.573113:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765435.573115:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765435.573120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2267 00000100:00080000:3.0:1775765435.573125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765435.573130:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765435.573140:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765435.573153:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066286592/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765435.573163:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765435.573166:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765435.573168:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765435.573172:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2267 00000100:00080000:3.0:1775765435.573175:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765435.573178:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765435.573182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765435.573191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066286720/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765435.573199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765435.573202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765435.573204:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765435.573207:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2267 00000100:00080000:3.0:1775765435.573210:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765435.573213:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765435.573218:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765435.573226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066286848/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765435.573235:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765435.573238:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765435.573240:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765435.573244:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2267 00000100:00080000:3.0:1775765435.573247:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765435.573251:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765435.573254:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765435.573262:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066286976/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765436.213068:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765437.237068:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765438.261039:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765439.285066:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765440.309092:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765440.692938:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765440.692958:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765440.692962:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765440.692972:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2272 00000100:00080000:3.0:1775765440.692981:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765440.692987:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765440.693019:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066287104/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765440.693048:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765440.693053:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765440.693055:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765440.693059:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2272 00000100:00080000:3.0:1775765440.693064:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765440.693069:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765440.693077:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765440.693090:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066287232/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765440.693105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765440.693107:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765440.693108:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765440.693112:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2272 00000100:00080000:3.0:1775765440.693114:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765440.693117:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765440.693119:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765440.693128:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066287360/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765440.693155:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765440.693158:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765440.693159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765440.693162:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2272 00000100:00080000:3.0:1775765440.693165:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765440.693171:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765440.693177:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765440.693187:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066287488/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765440.693199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765440.693202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765440.693203:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765440.693206:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2272 00000100:00080000:3.0:1775765440.693209:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765440.693214:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765440.693220:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765440.693239:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066287616/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765441.333033:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765442.357077:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765443.381145:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765444.405060:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765445.429055:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765445.812953:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765445.812961:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765445.812981:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765445.812990:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2277 00000100:00080000:3.0:1775765445.813000:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765445.813007:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765445.813041:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066287744/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765445.813070:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765445.813073:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765445.813075:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765445.813079:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2277 00000100:00080000:3.0:1775765445.813082:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765445.813084:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765445.813094:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765445.813103:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066287872/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765445.813112:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765445.813114:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765445.813115:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765445.813119:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2277 00000100:00080000:3.0:1775765445.813121:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765445.813124:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765445.813128:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765445.813134:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066288000/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765445.813143:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765445.813147:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765445.813152:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765445.813159:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2277 00000100:00080000:3.0:1775765445.813165:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765445.813169:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765445.813175:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765445.813189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066288128/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765445.813224:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765445.813228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765445.813231:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765445.813239:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2277 00000100:00080000:3.0:1775765445.813247:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765445.813253:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765445.813261:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765445.813294:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066288256/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765446.453015:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765447.477018:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765448.501073:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765449.525135:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765450.549142:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765450.933097:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765450.933110:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765450.933116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765450.933139:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2282 00000100:00080000:2.0:1775765450.933155:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765450.933166:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765450.933224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066288384/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765450.933298:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765450.933304:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765450.933307:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765450.933315:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2282 00000100:00080000:2.0:1775765450.933322:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765450.933330:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765450.933347:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765450.933375:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0380 x1862023066288512/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765450.933412:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765450.933417:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765450.933420:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765450.933427:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2282 00000100:00080000:2.0:1775765450.933433:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765450.933439:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765450.933447:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765450.933465:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1c00 x1862023066288640/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765450.933492:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765450.933497:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765450.933499:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765450.933506:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2282 00000100:00080000:2.0:1775765450.933524:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765450.933528:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765450.933535:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765450.933554:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066288768/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765450.933580:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765450.933593:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765450.933597:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765450.933613:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2282 00000100:00080000:2.0:1775765450.933622:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765450.933626:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765450.933633:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765450.933650:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066288896/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765451.573141:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765452.597130:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765453.621045:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765454.645038:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765455.669047:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765456.053023:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765456.053034:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765456.053050:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765456.053061:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2287 00000100:00080000:2.0:1775765456.053074:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765456.053084:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765456.053147:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066289024/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765456.053181:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765456.053186:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765456.053188:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765456.053193:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2287 00000100:00080000:2.0:1775765456.053198:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765456.053204:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765456.053228:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765456.053253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066289152/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765456.053273:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765456.053275:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765456.053277:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765456.053280:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2287 00000100:00080000:2.0:1775765456.053284:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765456.053286:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765456.053291:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765456.053302:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066289280/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765456.053309:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765456.053312:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765456.053313:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765456.053315:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2287 00000100:00080000:2.0:1775765456.053319:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765456.053321:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765456.053325:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765456.053336:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066289408/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765456.053344:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765456.053347:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765456.053350:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765456.053353:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2287 00000100:00080000:2.0:1775765456.053356:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765456.053358:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765456.053361:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765456.053367:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066289536/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1775765456.693060:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765457.717162:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765458.741159:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765459.765154:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1775765460.789147:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765461.173047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765461.173267:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066289664:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765461.173405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066289792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765461.173434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066289920:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765461.173449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290048:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765461.173464:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290176:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765461.813014:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765462.197132:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765462.837062:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765463.220945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765463.861061:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765464.244985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775765464.245327:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775765464.245397:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775765464.245425:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765464.245477:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765464.245497:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765464.245514:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765464.245528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765461/real 1775765464] req@ffff9d81901aed80 x1862023066289792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775765464.245542:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765464.245585:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066289792/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765464.245605:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765464.245614:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765464.245622:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066289792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765464.245637:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765461/real 1775765464] req@ffff9d81901af100 x1862023066289920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765464.245657:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066289920/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765464.245668:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765464.245673:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765464.245678:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066289920:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765464.245688:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765461/real 1775765464] req@ffff9d81901ad180 x1862023066290048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765464.245706:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066290048/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765464.245718:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765464.245723:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765464.245729:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290048:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765464.245739:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765461/real 1775765464] req@ffff9d81901afb80 x1862023066290176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765464.245756:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066290176/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765464.245769:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765464.245774:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765464.245778:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290176:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765464.245788:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765461/real 1775765464] req@ffff9d81901ae680 x1862023066289664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765464.245806:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066289664/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765464.245816:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765464.245821:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765464.245826:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066289664:192.168.202.105@tcp:250:kworker.0 00010000:00010000:1.0:1775765464.885140:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765465.268974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765465.909210:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765466.292945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765466.293429:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066290304:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765466.293520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765466.293650:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765466/real 1775765466] req@ffff9d81901af800 x1862023066290304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765466.293679:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066290304/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765466.293692:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765466.293697:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765466.293702:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066290304:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765466.293712:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765466.293739:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765466.293753:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765466/real 1775765466] req@ffff9d81901ae680 x1862023066290432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765466.293774:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066290432/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765466.293783:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765466.293787:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765466.293791:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765466.293798:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290560:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765466.293820:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765466.293830:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765466/real 1775765466] req@ffff9d81901afb80 x1862023066290560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765466.293839:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066290560/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765466.293845:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765466.293848:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765466.293850:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290560:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765466.293853:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765466.293872:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765466.293916:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765466/real 1775765466] req@ffff9d81901ad180 x1862023066290688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765466.293929:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066290688/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765466.293946:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765466.293950:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765466.293953:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290688:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765466.293957:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290816:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765466.293974:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765466.293981:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765466/real 1775765466] req@ffff9d81901af100 x1862023066290816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765466.293990:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066290816/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765466.294000:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765466.294003:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765466.294006:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066290816:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765466.933183:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765467.317017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765467.957167:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765468.340967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765468.981107:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765469.364964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765470.005193:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765470.388968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765471.029023:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765471.412979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765471.413119:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765471.413131:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765471.413137:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765471.413149:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2302 00000100:00080000:1.0:1775765471.413181:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765471.413190:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765471.413249:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553800 x1862023066290944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765471.413287:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765471.413292:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765471.413294:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765471.413298:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2302 00000100:00080000:1.0:1775765471.413303:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765471.413307:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765471.413335:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765471.413364:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066290944:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765471.413366:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066291072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765471.413397:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765471.413400:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765471.413402:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765471.413406:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2302 00000100:00080000:1.0:1775765471.413409:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765471.413415:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765471.413420:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765471.413444:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066291200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765471.413455:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765471.413459:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765471.413462:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765471.413465:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2302 00000100:00080000:1.0:1775765471.413478:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765471.413482:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765471.413486:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765471.413494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765471.413504:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066291328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765471.413515:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765471.413519:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765471.413522:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765471.413527:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2302 00000100:00080000:1.0:1775765471.413530:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765471.413534:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765471.413540:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765471.413556:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066291456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765471.413568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765471/real 1775765471] req@ffff9d8190553800 x1862023066290944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765471.413586:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553800 x1862023066290944/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765471.413618:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765471.413625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765471.413630:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066290944:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765471.413661:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765471.413705:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765471.413739:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765471/real 1775765471] req@ffff9d8190551500 x1862023066291072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765471.413789:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066291072/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765471.413799:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765471.413803:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765471.413806:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765471.413814:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291200:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765471.413831:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765471.413839:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765471/real 1775765471] req@ffff9d8190553480 x1862023066291200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765471.413847:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066291200/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765471.413852:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765471.413855:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765471.413858:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291200:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765471.413862:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765471.413875:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765471.413880:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765471/real 1775765471] req@ffff9d8190553100 x1862023066291328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765471.413907:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066291328/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765471.413914:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765471.413916:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765471.413918:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291328:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765471.413935:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291456:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765471.413953:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765471.413964:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765471/real 1775765471] req@ffff9d8190550e00 x1862023066291456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765471.413977:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066291456/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765471.413986:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765471.413990:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765471.413994:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291456:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765472.053141:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765472.437120:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765473.077080:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765473.460956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765474.101253:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765474.484971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765475.125012:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765475.508947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765476.149138:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765476.532966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765476.533181:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066291584:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765476.533255:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765476.533347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765476/real 1775765476] req@ffff9d81901aed80 x1862023066291584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765476.533390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066291584/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765476.533407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765476.533415:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765476.533420:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066291584:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765476.533444:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765476.533466:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765476.533478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765476/real 1775765476] req@ffff9d81901af100 x1862023066291712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765476.533488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066291712/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765476.533495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765476.533498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765476.533501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765476.533506:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291840:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765476.533522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765476.533529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765476/real 1775765476] req@ffff9d81901ad180 x1862023066291840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765476.533539:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066291840/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765476.533562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765476.533565:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765476.533567:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291840:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765476.533572:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765476.533587:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765476.533595:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765476/real 1775765476] req@ffff9d81901afb80 x1862023066291968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765476.533605:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066291968/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765476.533613:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765476.533615:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765476.533617:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066291968:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765476.533622:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292096:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765476.533634:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765476.533641:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765476/real 1775765476] req@ffff9d81901ae680 x1862023066292096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765476.533657:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066292096/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765476.533688:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765476.533691:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765476.533696:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292096:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765477.173079:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765477.556951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765478.197123:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765478.580967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765479.221048:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765479.604960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765480.245082:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765480.628939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765481.269232:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765481.652944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765481.653982:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765481.653988:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765481.653992:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765481.653999:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2313 00000100:00080000:2.0:1775765481.654013:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765481.654020:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765481.654058:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066292224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765481.654085:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765481.654089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765481.654091:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765481.654096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2313 00000100:00080000:2.0:1775765481.654100:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765481.654105:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765481.654114:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765481.654128:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066292352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765481.654128:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066292224:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765481.654139:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765481.654142:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765481.654144:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765481.654148:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2313 00000100:00080000:2.0:1775765481.654151:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765481.654154:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765481.654158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765481.654167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066292480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765481.654176:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765481.654180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765481.654181:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775765481.654183:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765481.654185:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2313 00000100:00080000:2.0:1775765481.654188:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765481.654192:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765481.654196:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765481.654206:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066292608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765481.654215:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765481.654217:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765481.654219:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765481.654223:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2313 00000100:00100000:3.0:1775765481.654225:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765481/real 1775765481] req@ffff9d81984a0a80 x1862023066292224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765481.654226:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765481.654230:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765481.654234:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765481.654242:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066292736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765481.654243:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066292224/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765481.654254:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765481.654258:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765481.654262:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066292224:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765481.654283:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765481.654305:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765481.654315:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765481/real 1775765481] req@ffff9d81984a0e00 x1862023066292352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765481.654324:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066292352/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765481.654330:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765481.654332:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765481.654335:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765481.654338:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292480:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765481.654349:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765481.654355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765481/real 1775765481] req@ffff9d81984a3100 x1862023066292480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765481.654363:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066292480/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765481.654368:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765481.654369:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765481.654371:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292480:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765481.654374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765481.654384:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765481.654389:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765481/real 1775765481] req@ffff9d81984a3800 x1862023066292608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765481.654396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066292608/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765481.654407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765481.654409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765481.654411:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292608:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765481.654414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292736:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765481.654423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765481.654428:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765481/real 1775765481] req@ffff9d81984a1880 x1862023066292736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765481.654434:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066292736/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765481.654439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765481.654441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765481.654443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292736:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765482.293058:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765482.676994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765483.317156:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765483.701072:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765484.341064:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765484.725001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765485.365087:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765485.749057:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765486.389090:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765486.773013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765486.773057:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765486.773066:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765486.773070:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765486.773080:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2318 00000100:00080000:1.0:1775765486.773091:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765486.773098:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765486.773159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066292864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765486.773202:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765486.773208:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765486.773211:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765486.773218:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2318 00000100:00080000:1.0:1775765486.773227:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765486.773235:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765486.773251:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765486.773257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066292864:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765486.773298:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066292992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765486.773326:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765486.773331:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765486.773335:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765486.773342:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2318 00000800:00000100:3.0:1775765486.773349:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775765486.773350:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765486.773360:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765486.773369:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765486.773391:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066293120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765486.773407:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765486.773415:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1775765486.773416:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765486/real 1775765486] req@ffff9d8190551880 x1862023066292864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765486.773420:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765486.773435:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2318 00000100:00080000:1.0:1775765486.773446:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765486.773454:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765486.773463:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765486.773464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066292864/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765486.773482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775765486.773490:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066293248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765486.773490:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765486.773498:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066292864:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765486.773519:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765486.773523:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765486.773525:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765486.773553:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2318 00000100:00100000:3.0:1775765486.773553:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292992:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775765486.773563:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765486.773568:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765486.773575:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765486.773597:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066293376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765486.773623:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765486.773662:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765486/real 1775765486] req@ffff9d8190550e00 x1862023066292992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765486.773676:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066292992/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765486.773684:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765486.773687:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765486.773691:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066292992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765486.773697:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293120:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765486.773714:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765486.773723:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765486/real 1775765486] req@ffff9d8190553100 x1862023066293120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765486.773734:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066293120/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765486.773742:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765486.773744:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765486.773748:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293120:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765486.773754:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765486.773769:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765486.773777:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765486/real 1775765486] req@ffff9d8190553480 x1862023066293248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765486.773787:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066293248/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765486.773796:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765486.773798:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765486.773801:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293248:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765486.773818:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293376:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765486.773845:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765486.773860:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765486/real 1775765486] req@ffff9d8190551500 x1862023066293376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765486.773879:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066293376/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765486.773945:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765486.773950:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765486.773954:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293376:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765487.413033:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765487.797038:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765488.437064:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765488.821052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765489.461037:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765489.844971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765490.485043:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765490.868947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765491.509067:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765491.892952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765491.893093:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066293504:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765491.893151:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765491.893192:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765491/real 1775765491] req@ffff9d81901af800 x1862023066293504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765491.893228:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066293504/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765491.893236:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765491.893240:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765491.893243:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066293504:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765491.893259:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765491.893284:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765491.893293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765491/real 1775765491] req@ffff9d81901ae680 x1862023066293632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765491.893301:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066293632/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765491.893307:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765491.893309:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765491.893311:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765491.893315:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293760:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765491.893328:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765491.893333:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765491/real 1775765491] req@ffff9d81901afb80 x1862023066293760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765491.893341:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066293760/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765491.893346:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765491.893348:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765491.893350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293760:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765491.893353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765491.893364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765491.893370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765491/real 1775765491] req@ffff9d81901ad180 x1862023066293888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765491.893377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066293888/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765491.893383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765491.893385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765491.893387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066293888:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765491.893390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294016:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765491.893399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765491.893405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765491/real 1775765491] req@ffff9d81901af100 x1862023066294016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765491.893417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066294016/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765491.893422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765491.893424:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765491.893426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294016:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765492.533043:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765492.916967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765493.557143:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765493.941007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765494.581009:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765494.964940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765495.605118:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765495.988995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765496.629116:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765497.013086:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765497.013371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066294144:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765497.013489:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765497.013591:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765497/real 1775765497] req@ffff9d81901aed80 x1862023066294144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765497.013629:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066294144/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765497.013652:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765497.013657:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765497.013662:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066294144:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765497.013673:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765497.013713:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765497.013730:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765497/real 1775765497] req@ffff9d81901af100 x1862023066294272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765497.013746:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066294272/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765497.013756:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765497.013777:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765497.013781:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765497.013942:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294400:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765497.013973:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765497.013989:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765497/real 1775765497] req@ffff9d81901ad180 x1862023066294400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765497.014005:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066294400/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765497.014034:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765497.014038:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765497.014042:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294400:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765497.014050:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765497.014071:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765497.014081:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765497/real 1775765497] req@ffff9d81901afb80 x1862023066294528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765497.014094:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066294528/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765497.014114:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765497.014117:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765497.014121:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294528:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765497.014127:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294656:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765497.014150:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765497.014159:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765497/real 1775765497] req@ffff9d81901ae680 x1862023066294656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765497.014172:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066294656/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765497.014181:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765497.014184:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765497.014187:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294656:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765497.653196:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765498.037018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765498.677079:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765499.060982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765499.701175:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765500.085009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765500.725054:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765501.108974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765501.749022:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765502.132940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765502.132960:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765502.132965:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765502.132970:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765502.132978:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2333 00000100:00080000:2.0:1775765502.132987:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765502.132993:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765502.133025:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066294784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765502.133059:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765502.133062:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765502.133063:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765502.133066:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2333 00000100:00080000:2.0:1775765502.133075:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765502.133080:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765502.133090:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765502.133107:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066294784:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765502.133111:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066294912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765502.133125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765502.133132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765502.133134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765502.133137:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2333 00000100:00080000:2.0:1775765502.133142:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765502.133145:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765502.133150:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765502.133159:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066295040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765502.133169:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775765502.133173:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765502.133175:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765502.133178:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765502.133184:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2333 00000100:00080000:2.0:1775765502.133189:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765502.133193:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765502.133199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765502.133214:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066295168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765502.133225:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765502.133227:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765502.133229:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765502.133234:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2333 00000100:00100000:3.0:1775765502.133235:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765502/real 1775765502] req@ffff9d81984a2680 x1862023066294784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765502.133238:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765502.133245:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765502.133253:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765502.133262:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066294784/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765502.133267:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066295296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765502.133289:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765502.133296:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765502.133300:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066294784:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765502.133318:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765502.133336:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765502.133345:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765502/real 1775765502] req@ffff9d81984a1500 x1862023066294912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765502.133353:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066294912/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765502.133359:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765502.133361:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765502.133363:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066294912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765502.133367:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295040:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765502.133377:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765502.133382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765502/real 1775765502] req@ffff9d81984a1180 x1862023066295040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765502.133390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066295040/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765502.133395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765502.133397:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765502.133399:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295040:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765502.133403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765502.133414:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765502.133419:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765502/real 1775765502] req@ffff9d81984a1f80 x1862023066295168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765502.133426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066295168/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765502.133432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765502.133434:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765502.133436:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295168:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765502.133439:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295296:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765502.133449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765502.133454:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765502/real 1775765502] req@ffff9d81984a2300 x1862023066295296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765502.133461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066295296/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765502.133467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765502.133468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765502.133470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295296:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765502.773052:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765503.156998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765503.797058:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765504.180959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765504.821037:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765505.205006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765505.845101:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765506.228959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765506.869054:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765507.252934:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765507.253126:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066295424:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765507.253231:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765507.253287:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765507/real 1775765507] req@ffff9d81901af800 x1862023066295424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765507.253304:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066295424/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765507.253313:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765507.253317:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765507.253320:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066295424:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765507.253344:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765507.253363:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765507.253371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765507/real 1775765507] req@ffff9d81901ae680 x1862023066295552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765507.253380:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066295552/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765507.253386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765507.253388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765507.253391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765507.253394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295680:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765507.253408:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765507.253414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765507/real 1775765507] req@ffff9d81901afb80 x1862023066295680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765507.253422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066295680/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765507.253436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765507.253438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765507.253439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295680:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765507.253443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765507.253454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765507.253459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765507/real 1775765507] req@ffff9d81901ad180 x1862023066295808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765507.253466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066295808/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765507.253474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765507.253475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765507.253477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295808:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765507.253480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295936:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765507.253490:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765507.253494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765507/real 1775765507] req@ffff9d81901af100 x1862023066295936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765507.253506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066295936/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765507.253522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765507.253524:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765507.253526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066295936:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765507.893160:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765508.276948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765508.917198:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765509.301011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765509.941101:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765510.324964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765510.965194:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765511.349093:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765511.989114:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765512.372981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765512.373294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066296064:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765512.373403:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765512.373460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765512/real 1775765512] req@ffff9d81901aed80 x1862023066296064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765512.373516:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066296064/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765512.373549:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765512.373560:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765512.373567:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066296064:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765512.373584:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765512.373647:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765512.373662:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765512/real 1775765512] req@ffff9d81901af100 x1862023066296192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765512.373724:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066296192/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765512.373733:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765512.373736:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765512.373739:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765512.373785:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296320:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765512.373806:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765512.373817:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765512/real 1775765512] req@ffff9d81901ad180 x1862023066296320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765512.373829:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066296320/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765512.373838:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765512.373841:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765512.373844:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296320:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765512.373849:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765512.373864:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765512.373872:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765512/real 1775765512] req@ffff9d81901afb80 x1862023066296448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765512.373916:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066296448/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765512.373932:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765512.373936:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765512.373939:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296448:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765512.373945:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296576:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765512.373963:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765512.373973:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765512/real 1775765512] req@ffff9d81901ae680 x1862023066296576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765512.373984:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066296576/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765512.373992:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765512.373995:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765512.373998:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296576:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765513.013107:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765513.397062:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765514.037109:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765514.420963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765515.061168:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765515.445067:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765516.085041:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765516.468991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765517.109219:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765517.492969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765517.493376:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066296704:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765517.493460:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765517.493521:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765517/real 1775765517] req@ffff9d81901af800 x1862023066296704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765517.493581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af800 x1862023066296704/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765517.493609:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765517.493621:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765517.493655:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066296704:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765517.493699:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765517.493743:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765517.493765:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765517/real 1775765517] req@ffff9d81901ae680 x1862023066296832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765517.493794:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066296832/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765517.493817:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765517.493836:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765517.493846:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765517.493855:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296960:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765517.493924:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765517.493940:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765517/real 1775765517] req@ffff9d81901afb80 x1862023066296960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765517.494009:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066296960/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765517.494022:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765517.494028:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765517.494033:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066296960:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765517.494054:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765517.494080:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765517.494092:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765517/real 1775765517] req@ffff9d81901ad180 x1862023066297088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765517.494116:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066297088/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765517.494126:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765517.494130:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765517.494133:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297088:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765517.494140:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297216:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765517.494158:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765517.494168:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765517/real 1775765517] req@ffff9d81901af100 x1862023066297216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765517.494181:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066297216/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765517.494190:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765517.494194:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765517.494197:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297216:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765518.133068:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765518.516945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765519.157152:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765519.540950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765520.181104:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765520.564964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765521.205076:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765521.588987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765522.229088:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765522.613033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765522.613265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066297344:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765522.613353:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765522.613435:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765522/real 1775765522] req@ffff9d81901aed80 x1862023066297344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765522.613476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066297344/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765522.613492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765522.613498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765522.613504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066297344:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765522.613537:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765522.613568:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765522.613601:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765522/real 1775765522] req@ffff9d81901af100 x1862023066297472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765522.613618:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066297472/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765522.613630:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765522.613635:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765522.613640:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765522.613648:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297600:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765522.613673:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765522.613684:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765522/real 1775765522] req@ffff9d81901ad180 x1862023066297600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765522.613695:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066297600/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765522.613708:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765522.613711:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765522.613714:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297600:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765522.613719:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765522.613734:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765522.613741:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765522/real 1775765522] req@ffff9d81901afb80 x1862023066297728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765522.613750:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066297728/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765522.613759:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765522.613761:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765522.613763:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297728:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765522.613768:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297856:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765522.613781:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765522.613798:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765522/real 1775765522] req@ffff9d81901ae680 x1862023066297856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765522.613808:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066297856/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765522.613814:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765522.613816:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765522.613836:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066297856:192.168.202.105@tcp:8:kworker.0 00010000:00010000:1.0:1775765523.253011:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765523.637089:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765524.277064:0:33224:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765524.661044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:1.0:1775765525.301120:0:33225:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765525.685022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:2.0:1775765526.325053:0:34145:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 90/90 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1775765526.325118:0:33224:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 90 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1775765526.325129:0:33224:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800028400020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775765526.325163:0:34145:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 85/85 locks to regular blp list, 1 blwis in pool 00000008:00010000:1.0:1775765526.325213:0:33224:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800029400020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1775765526.325234:0:33224:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800029400020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1775765526.325234:0:33225:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 85 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1775765526.325277:0:34145:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 86/86 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1775765526.325411:0:34145:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 78/164 locks to regular blp list, 2 blwis in pool 00010000:00010000:1.0:1775765526.325434:0:33224:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 4/0,0 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800029400020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1775765526.325567:0:34195:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 86 locks of 78 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1775765526.325592:0:33225:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1775765526.325659:0:33225:0:(osc_cache.c:2012:try_to_add_extent_for_io()) ### extent: 000000002b6030ac ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:1.0:1775765526.325667:0:33224:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d8190553800 - object 0x280000400:552 offset 0<>313063 00000008:00010000:2.0:1775765526.325669:0:33225:0:(osc_cache.c:1167:osc_extent_make_ready()) ### extent: 000000002b6030ac ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775765526.325700:0:33225:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00100000:2.0:1775765526.325767:0:33225:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9d81984a1c00 - object 0x2c0000400:551 offset 0<>1765 00010000:00010000:3.0:1775765526.325771:0:34196:0:(ldlm_lockd.c:2794:ldlm_bl_get_work()) Got 78 locks of 0 total in blp. (0 blwis in pool) 00000100:00100000:3.0:1775765526.325817:0:10944:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:3.0:1775765526.325830:0:10944:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9d81984a1c00 x1862023066298112/t0(0) o4->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:6/4 lens 2256/448 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'cp.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1775765526.325984:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1775765526.325995:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1775765526.325998:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765526.326006:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1775765526.326049:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x4e:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326075:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326097:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x4d:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326146:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326168:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x4c:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326203:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326229:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x4b:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326284:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326303:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x4a:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326320:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326342:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x49:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326419:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326446:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x48:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326491:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326520:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x47:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326677:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326751:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x46:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326803:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326826:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x45:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326860:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326906:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x44:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326927:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326948:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x43:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.326977:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.326998:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x42:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327031:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327058:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x41:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327088:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327114:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x40:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327138:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327163:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x3f:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327188:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327214:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x3e:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327239:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327259:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x3d:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327307:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327385:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x3c:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327420:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327438:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x3b:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327463:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327489:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x3a:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327519:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327556:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x39:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327580:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327600:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x38:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327639:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327695:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x37:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327731:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9200/0xeef44110eff5494 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155484 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327748:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x36:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327780:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5800/0xeef44110eff5471 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155429 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327802:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x35:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327835:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b200/0xeef44110eff544e lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321553ce expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327856:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x34:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327903:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b000/0xeef44110eff542b lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155373 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327929:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x33:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.327954:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b800/0xeef44110eff5408 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155318 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.327975:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x32:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328009:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1ae00/0xeef44110eff53e5 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321552bd expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328030:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x31:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328066:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8180030c00/0xeef44110eff53c2 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155262 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328088:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x30:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328118:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a600/0xeef44110eff539f lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155207 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328139:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x2f:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328169:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a400/0xeef44110eff537c lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321551ac expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328192:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x2e:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328218:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4800/0xeef44110eff5359 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155151 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328240:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x2d:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328272:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8c00/0xeef44110eff5336 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321550f6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328320:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x2c:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328371:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5400/0xeef44110eff5313 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215509b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328386:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x2b:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328413:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5200/0xeef44110eff52f0 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32155040 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328436:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x2a:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328465:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4c00/0xeef44110eff52cd lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154fe5 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328489:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x29:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328517:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5000/0xeef44110eff52aa lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154f8a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328551:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x28:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328575:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5600/0xeef44110eff5287 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154f2f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328596:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x27:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328631:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5e00/0xeef44110eff5264 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154ed4 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328652:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x26:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328688:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5a00/0xeef44110eff5241 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154e79 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328715:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x25:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328744:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4200/0xeef44110eff521e lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154e1e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328765:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x24:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328792:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9600/0xeef44110eff51fb lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154dc3 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328812:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x23:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328846:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9800/0xeef44110eff51d8 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154d68 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328866:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x22:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.328919:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8400/0xeef44110eff51b5 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154d0d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.328978:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x21:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329002:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8800/0xeef44110eff5192 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154cb2 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329025:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x20:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329051:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9400/0xeef44110eff516f lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154c57 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329075:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x1f:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329102:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff514c lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154bfc expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329127:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x1e:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329158:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9e00/0xeef44110eff5129 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154ba1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329180:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x1d:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329214:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9600/0xeef44110eff5106 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154b46 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329234:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x1c:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329260:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186cccc00/0xeef44110eff50e3 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154aeb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329280:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x1b:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329319:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccde00/0xeef44110eff50c0 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154a90 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329377:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x1a:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329436:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9c00/0xeef44110eff509d lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154a35 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329450:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x19:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329481:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8200/0xeef44110eff507a lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321549da expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329502:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x18:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329529:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8000/0xeef44110eff5057 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215497f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329567:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x17:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329605:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9e00/0xeef44110eff5034 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154924 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329628:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x16:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329660:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8800/0xeef44110eff5011 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321548c9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329721:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x15:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329785:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9800/0xeef44110eff4fee lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215486e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329858:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x14:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329913:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8c00/0xeef44110eff4fcb lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154813 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329933:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x13:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.329957:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd400/0xeef44110eff4fa8 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321547b8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.329978:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x12:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330013:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059ea00/0xeef44110eff4f85 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215475d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330035:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x11:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330068:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccce00/0xeef44110eff4f62 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154702 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330100:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x10:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330123:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd000/0xeef44110eff4f3f lrc: 2/0,0 mode: PR/PR res: [0x240004281:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321546a7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330149:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0xf:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330173:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccc400/0xeef44110eff4f1c lrc: 2/0,0 mode: PR/PR res: [0x240004281:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215464c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330200:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0xe:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330225:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd200/0xeef44110eff4ef9 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321545f1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330252:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0xd:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330277:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd600/0xeef44110eff4ed6 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154596 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330313:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0xc:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330366:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccc600/0xeef44110eff4eb3 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215453b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330396:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0xb:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330413:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819841b200/0xeef44110eff4e90 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321544e0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330435:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0xa:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330464:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059ee00/0xeef44110eff4e6d lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154485 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330488:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x9:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330517:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f400/0xeef44110eff4e4a lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa3215442a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330551:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x8:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330580:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059e600/0xeef44110eff4e27 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321543cf expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330607:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x7:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330631:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f000/0xeef44110eff4e04 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154374 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330659:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x6:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330683:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f200/0xeef44110eff4de1 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154319 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330710:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330735:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f600/0xeef44110eff4dbe lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321542be expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330761:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x4:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330785:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81876ad400/0xeef44110eff4d9b lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa32154263 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330809:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x3:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330833:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190fca800/0xeef44110eff4d6a lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4dcde3aa321541de expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1775765526.330858:0:34196:0:(vvp_object.c:117:vvp_conf_set()) [0x240004281:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1775765526.330916:0:34196:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f62400/0xeef44110eff4d86 lrc: 2/0,0 mode: PR/PR res: [0x240004281:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4dcde3aa3215421d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775765526.330965:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 1/0,0 mode: --/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1775765526.330972:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.330979:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.330984:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.330989:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.330995:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 1/0,0 mode: --/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331000:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 1/0,0 mode: --/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331006:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331021:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331026:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331032:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 1/0,0 mode: --/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331039:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 1/0,0 mode: --/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331044:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331056:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331063:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331072:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 1/0,0 mode: --/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331079:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 1/0,0 mode: --/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331084:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331105:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331110:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331119:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331125:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 1/0,0 mode: --/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331139:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 1/0,0 mode: --/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331145:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331153:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9200/0xeef44110eff5494 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155484 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331159:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5800/0xeef44110eff5471 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155429 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331164:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b200/0xeef44110eff544e lrc: 1/0,0 mode: --/PR res: [0x240004281:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321553ce expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331172:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b000/0xeef44110eff542b lrc: 1/0,0 mode: --/PR res: [0x240004281:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155373 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331182:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b800/0xeef44110eff5408 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155318 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331188:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1ae00/0xeef44110eff53e5 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321552bd expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331209:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8180030c00/0xeef44110eff53c2 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155262 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331214:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a600/0xeef44110eff539f lrc: 1/0,0 mode: --/PR res: [0x240004281:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155207 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331219:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a400/0xeef44110eff537c lrc: 1/0,0 mode: --/PR res: [0x240004281:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321551ac expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331225:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4800/0xeef44110eff5359 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155151 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331230:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8c00/0xeef44110eff5336 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321550f6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331242:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5400/0xeef44110eff5313 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215509b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331255:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5200/0xeef44110eff52f0 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155040 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331260:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4c00/0xeef44110eff52cd lrc: 1/0,0 mode: --/PR res: [0x240004281:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154fe5 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331265:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5000/0xeef44110eff52aa lrc: 1/0,0 mode: --/PR res: [0x240004281:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154f8a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331270:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5600/0xeef44110eff5287 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154f2f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331275:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5e00/0xeef44110eff5264 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154ed4 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331280:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5a00/0xeef44110eff5241 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154e79 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331286:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4200/0xeef44110eff521e lrc: 1/0,0 mode: --/PR res: [0x240004281:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154e1e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331302:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9600/0xeef44110eff51fb lrc: 1/0,0 mode: --/PR res: [0x240004281:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154dc3 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331311:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9800/0xeef44110eff51d8 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154d68 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331321:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8400/0xeef44110eff51b5 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154d0d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331330:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8800/0xeef44110eff5192 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154cb2 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331344:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9400/0xeef44110eff516f lrc: 1/0,0 mode: --/PR res: [0x240004281:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154c57 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331363:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff514c lrc: 1/0,0 mode: --/PR res: [0x240004281:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154bfc expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331385:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9e00/0xeef44110eff5129 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154ba1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331411:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9600/0xeef44110eff5106 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154b46 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331437:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186cccc00/0xeef44110eff50e3 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154aeb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331457:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccde00/0xeef44110eff50c0 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154a90 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331479:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9c00/0xeef44110eff509d lrc: 1/0,0 mode: --/PR res: [0x240004281:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154a35 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331503:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8200/0xeef44110eff507a lrc: 1/0,0 mode: --/PR res: [0x240004281:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321549da expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331528:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8000/0xeef44110eff5057 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215497f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331566:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9e00/0xeef44110eff5034 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154924 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331594:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8800/0xeef44110eff5011 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321548c9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331622:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9800/0xeef44110eff4fee lrc: 1/0,0 mode: --/PR res: [0x240004281:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215486e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331650:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8c00/0xeef44110eff4fcb lrc: 1/0,0 mode: --/PR res: [0x240004281:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154813 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331675:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd400/0xeef44110eff4fa8 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321547b8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331690:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059ea00/0xeef44110eff4f85 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215475d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331708:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccce00/0xeef44110eff4f62 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154702 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331728:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd000/0xeef44110eff4f3f lrc: 1/0,0 mode: --/PR res: [0x240004281:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321546a7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331748:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccc400/0xeef44110eff4f1c lrc: 1/0,0 mode: --/PR res: [0x240004281:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215464c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331768:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd200/0xeef44110eff4ef9 lrc: 1/0,0 mode: --/PR res: [0x240004281:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321545f1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331788:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd600/0xeef44110eff4ed6 lrc: 1/0,0 mode: --/PR res: [0x240004281:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154596 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331808:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccc600/0xeef44110eff4eb3 lrc: 1/0,0 mode: --/PR res: [0x240004281:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215453b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331828:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819841b200/0xeef44110eff4e90 lrc: 1/0,0 mode: --/PR res: [0x240004281:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321544e0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331848:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059ee00/0xeef44110eff4e6d lrc: 1/0,0 mode: --/PR res: [0x240004281:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154485 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331870:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f400/0xeef44110eff4e4a lrc: 1/0,0 mode: --/PR res: [0x240004281:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215442a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331894:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059e600/0xeef44110eff4e27 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321543cf expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331909:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f000/0xeef44110eff4e04 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154374 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331919:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f200/0xeef44110eff4de1 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154319 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331945:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f600/0xeef44110eff4dbe lrc: 1/0,0 mode: --/PR res: [0x240004281:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321542be expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331969:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81876ad400/0xeef44110eff4d9b lrc: 1/0,0 mode: --/PR res: [0x240004281:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154263 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.331993:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190fca800/0xeef44110eff4d6a lrc: 1/0,0 mode: --/PR res: [0x240004281:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321541de expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332017:0:34196:0:(ldlm_request.c:1438:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f62400/0xeef44110eff4d86 lrc: 1/0,0 mode: --/PR res: [0x240004281:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215421d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775765526.332039:0:34196:0:(ldlm_request.c:1490:ldlm_cancel_pack()) 78 locks packed 00010000:00010000:2.0:1775765526.332061:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365600/0xeef44110eff57dc lrc: 0/0,0 mode: --/PR res: [0x240004281:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155d0c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00080000:1.0:1775765526.332093:0:10946:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9d81984a0380 x1862023066298240/t0(0) o103->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:17/18 lens 936/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1775765526.332093:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a200/0xeef44110eff57b9 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155cb1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332114:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364400/0xeef44110eff5796 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155c56 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332142:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190364c00/0xeef44110eff5773 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155bfb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332176:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365e00/0xeef44110eff5750 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155ba0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332206:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365400/0xeef44110eff572d lrc: 0/0,0 mode: --/PR res: [0x240004281:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155b45 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332218:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190365000/0xeef44110eff570a lrc: 0/0,0 mode: --/PR res: [0x240004281:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155aea expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332241:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1bc00/0xeef44110eff56e7 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155a8f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332264:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8c00/0xeef44110eff56c4 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155a34 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332286:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583b200/0xeef44110eff56a1 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321559d9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332305:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a600/0xeef44110eff567e lrc: 0/0,0 mode: --/PR res: [0x240004281:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215597e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332352:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a800/0xeef44110eff565b lrc: 0/0,0 mode: --/PR res: [0x240004281:0x43:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155923 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332375:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a400/0xeef44110eff5638 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x42:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321558c8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332388:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583a200/0xeef44110eff5615 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x41:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215586d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332413:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d818583ae00/0xeef44110eff55f2 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x40:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155812 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332429:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8200/0xeef44110eff55cf lrc: 0/0,0 mode: --/PR res: [0x240004281:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321557b7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332450:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b600/0xeef44110eff55ac lrc: 0/0,0 mode: --/PR res: [0x240004281:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215575c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332468:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9400/0xeef44110eff5589 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155701 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332486:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4400/0xeef44110eff5566 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321556a6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332511:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8600/0xeef44110eff5543 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215564b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332551:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d8000/0xeef44110eff5520 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321555f0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332566:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9c00/0xeef44110eff54fd lrc: 0/0,0 mode: --/PR res: [0x240004281:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155595 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332602:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9600/0xeef44110eff54da lrc: 0/0,0 mode: --/PR res: [0x240004281:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215553a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332619:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9800/0xeef44110eff54b7 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321554df expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332637:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81903d9200/0xeef44110eff5494 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155484 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332655:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5800/0xeef44110eff5471 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155429 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332670:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b200/0xeef44110eff544e lrc: 0/0,0 mode: --/PR res: [0x240004281:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321553ce expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332691:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b000/0xeef44110eff542b lrc: 0/0,0 mode: --/PR res: [0x240004281:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155373 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332711:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1b800/0xeef44110eff5408 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155318 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332750:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1ae00/0xeef44110eff53e5 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321552bd expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332766:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8180030c00/0xeef44110eff53c2 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155262 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332778:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a600/0xeef44110eff539f lrc: 0/0,0 mode: --/PR res: [0x240004281:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155207 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332786:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f1a400/0xeef44110eff537c lrc: 0/0,0 mode: --/PR res: [0x240004281:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321551ac expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332817:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4800/0xeef44110eff5359 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155151 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:1.0:1775765526.332820:0:10946:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000100:00080000:1.0:1775765526.332835:0:10946:0:(client.c:1830:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9d81901af800 x1862023066298368/t0(0) o103->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:17/18 lens 1000/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1775765526.332855:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8c00/0xeef44110eff5336 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321550f6 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332894:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5400/0xeef44110eff5313 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215509b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332911:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5200/0xeef44110eff52f0 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x2a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32155040 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332943:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4c00/0xeef44110eff52cd lrc: 0/0,0 mode: --/PR res: [0x240004281:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154fe5 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.332986:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5000/0xeef44110eff52aa lrc: 0/0,0 mode: --/PR res: [0x240004281:0x28:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154f8a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333002:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5600/0xeef44110eff5287 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154f2f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333021:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5e00/0xeef44110eff5264 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154ed4 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333050:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc5a00/0xeef44110eff5241 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154e79 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333085:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186dc4200/0xeef44110eff521e lrc: 0/0,0 mode: --/PR res: [0x240004281:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154e1e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333127:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9600/0xeef44110eff51fb lrc: 0/0,0 mode: --/PR res: [0x240004281:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154dc3 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333166:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9800/0xeef44110eff51d8 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154d68 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333192:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8400/0xeef44110eff51b5 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154d0d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333205:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb8800/0xeef44110eff5192 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154cb2 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333223:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9400/0xeef44110eff516f lrc: 0/0,0 mode: --/PR res: [0x240004281:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154c57 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333237:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190fcb200/0xeef44110eff514c lrc: 0/0,0 mode: --/PR res: [0x240004281:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154bfc expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333272:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9e00/0xeef44110eff5129 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154ba1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333288:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8188fb9600/0xeef44110eff5106 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154b46 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333302:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186cccc00/0xeef44110eff50e3 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154aeb expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333318:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccde00/0xeef44110eff50c0 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x1a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154a90 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333343:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9c00/0xeef44110eff509d lrc: 0/0,0 mode: --/PR res: [0x240004281:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154a35 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333378:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8200/0xeef44110eff507a lrc: 0/0,0 mode: --/PR res: [0x240004281:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321549da expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333418:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8000/0xeef44110eff5057 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215497f expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333474:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9e00/0xeef44110eff5034 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x16:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154924 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333512:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8800/0xeef44110eff5011 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321548c9 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333561:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b9800/0xeef44110eff4fee lrc: 0/0,0 mode: --/PR res: [0x240004281:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215486e expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333582:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81902b8c00/0xeef44110eff4fcb lrc: 0/0,0 mode: --/PR res: [0x240004281:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154813 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333596:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd400/0xeef44110eff4fa8 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321547b8 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333622:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059ea00/0xeef44110eff4f85 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215475d expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333653:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccce00/0xeef44110eff4f62 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154702 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333665:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd000/0xeef44110eff4f3f lrc: 0/0,0 mode: --/PR res: [0x240004281:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321546a7 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333690:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccc400/0xeef44110eff4f1c lrc: 0/0,0 mode: --/PR res: [0x240004281:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215464c expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333732:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd200/0xeef44110eff4ef9 lrc: 0/0,0 mode: --/PR res: [0x240004281:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321545f1 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333759:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccd600/0xeef44110eff4ed6 lrc: 0/0,0 mode: --/PR res: [0x240004281:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154596 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333771:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8186ccc600/0xeef44110eff4eb3 lrc: 0/0,0 mode: --/PR res: [0x240004281:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215453b expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333791:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819841b200/0xeef44110eff4e90 lrc: 0/0,0 mode: --/PR res: [0x240004281:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321544e0 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333809:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059ee00/0xeef44110eff4e6d lrc: 0/0,0 mode: --/PR res: [0x240004281:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154485 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333836:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f400/0xeef44110eff4e4a lrc: 0/0,0 mode: --/PR res: [0x240004281:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa3215442a expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333850:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059e600/0xeef44110eff4e27 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321543cf expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333896:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f000/0xeef44110eff4e04 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154374 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333908:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f200/0xeef44110eff4de1 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154319 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333932:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d819059f600/0xeef44110eff4dbe lrc: 0/0,0 mode: --/PR res: [0x240004281:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321542be expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333960:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d81876ad400/0xeef44110eff4d9b lrc: 0/0,0 mode: --/PR res: [0x240004281:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa32154263 expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.333997:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190fca800/0xeef44110eff4d6a lrc: 0/0,0 mode: --/PR res: [0x240004281:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4dcde3aa321541de expref: -99 pid: 34105 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1775765526.334035:0:34196:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9d81892c2000 lock: ffff9d8190f62400/0xeef44110eff4d86 lrc: 0/0,0 mode: --/PR res: [0x240004281:0x1:0x0].0x0 bits 0x10/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4dcde3aa3215421d expref: -99 pid: 34105 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1775765526.334073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765526.334076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1775765526.334080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765526.709077:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765527.732987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765527.733119:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066298496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765527.733279:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066298624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765527.733325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066298752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765527.733356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066298880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765527.733390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765528.373059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765528.756986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765529.780983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765530.804999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775765530.805194:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775765530.805252:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775765530.805271:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765530.805308:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765530.805316:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765530.805324:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765530.805335:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765530.805384:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765527/real 1775765530] req@ffff9d81901aed80 x1862023066299008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765530.805445:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066299008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765530.805470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765530.805475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765530.805480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299008:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765530.805489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765527/real 1775765530] req@ffff9d81901afb80 x1862023066298624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765530.805522:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066298624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765530.805532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765530.805536:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765530.805540:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066298624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765530.805549:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765527/real 1775765530] req@ffff9d81901ad180 x1862023066298752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765530.805563:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066298752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765530.805573:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765530.805576:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765530.805580:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066298752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765530.805588:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765527/real 1775765530] req@ffff9d81901af100 x1862023066298880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765530.805600:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066298880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765530.805610:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765530.805619:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765530.805622:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066298880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765530.805630:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765527/real 1775765530] req@ffff9d81901ae680 x1862023066298496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765530.805645:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066298496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765530.805652:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765530.805656:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765530.805660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066298496:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775765531.445098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765531.828960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765532.852953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765532.853119:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066299136:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765532.853201:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765532.853265:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765532/real 1775765532] req@ffff9d81901ae680 x1862023066299136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765532.853286:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066299136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765532.853301:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765532.853305:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765532.853308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066299136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765532.853328:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765532.853350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765532.853360:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765532/real 1775765532] req@ffff9d81901af100 x1862023066299264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765532.853382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066299264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765532.853389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765532.853392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765532.853395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765532.853400:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765532.853416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765532.853428:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765532/real 1775765532] req@ffff9d81901ad180 x1862023066299392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765532.853443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066299392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765532.853452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765532.853456:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765532.853461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765532.853494:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765532.853540:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765532.853560:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765532/real 1775765532] req@ffff9d81901afb80 x1862023066299520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765532.853580:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066299520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765532.853595:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765532.853599:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765532.853604:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765532.853614:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765532.853648:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765532.853678:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765532/real 1775765532] req@ffff9d81901aed80 x1862023066299648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765532.853702:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066299648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765532.853714:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765532.853717:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765532.853720:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299648:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765533.877016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765534.517091:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765534.900928:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765535.925015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765536.948999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765537.589141:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765537.973049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765537.973289:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066299776:192.168.202.105@tcp:250:kworker.0 00000800:00000100:3.0:1775765537.973443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765537.973509:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765537/real 1775765537] req@ffff9d81901aed80 x1862023066299776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765537.973545:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765537.973555:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066299776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765537.973556:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2369 00000100:00080000:0.0:1775765537.973571:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765537.973580:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765537.973581:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775765537.973599:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:0.0:1775765537.973601:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765537.973612:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066299776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765537.973629:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066300288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765537.973651:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765537.973678:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765537.973693:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765537/real 1775765537] req@ffff9d81901afb80 x1862023066299904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765537.973702:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066299904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765537.973708:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765537.973712:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765537.973714:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066299904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765537.973721:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765537.973733:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765537.973739:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765537/real 1775765537] req@ffff9d81901ad180 x1862023066300032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765537.973747:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066300032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765537.973752:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765537.973754:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765537.973756:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765537.973760:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765537.973772:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765537.973777:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765537/real 1775765537] req@ffff9d81901af100 x1862023066300160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765537.973785:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066300160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765537.973796:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765537.973800:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765537.973804:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765537.973821:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765537.973843:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765537.973854:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765537/real 1775765537] req@ffff9d81901ae680 x1862023066300288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765537.973865:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066300288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765537.973872:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765537.973874:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765537.973904:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765538.613154:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765538.997048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765539.637144:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765540.021035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765540.661184:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765541.044985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765541.685074:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765542.068984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765542.709090:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765543.092989:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765543.092989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765543.092998:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765543.093002:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765543.093011:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2374 00000100:00080000:2.0:1775765543.093020:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765543.093027:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765543.093058:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066300416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765543.093079:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765543.093082:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765543.093083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765543.093086:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2374 00000100:00080000:2.0:1775765543.093089:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765543.093098:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765543.093106:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066300416:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775765543.093107:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765543.093119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066300544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765543.093127:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765543.093129:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765543.093131:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765543.093134:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2374 00000100:00080000:2.0:1775765543.093137:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765543.093139:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765543.093142:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765543.093149:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066300672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765543.093157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765543.093163:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775765543.093163:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765543.093168:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765543.093173:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2374 00000100:00080000:2.0:1775765543.093180:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765543.093184:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765543.093190:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765543.093206:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066300800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765543.093211:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765543/real 1775765543] req@ffff9d81984a0000 x1862023066300416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765543.093230:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765543.093233:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765543.093234:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765543.093239:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2374 00000100:00100000:3.0:1775765543.093240:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066300416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765543.093242:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765543.093250:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775765543.093253:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775765543.093256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775765543.093258:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765543.093263:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066300416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765543.093269:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066300928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765543.093286:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765543.093313:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765543.093327:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765543/real 1775765543] req@ffff9d81984a2a00 x1862023066300544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765543.093339:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066300544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765543.093347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765543.093350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765543.093354:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765543.093361:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765543.093393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765543.093403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765543/real 1775765543] req@ffff9d81984a2d80 x1862023066300672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765543.093411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066300672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765543.093417:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765543.093419:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765543.093421:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765543.093425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765543.093438:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765543.093444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765543/real 1775765543] req@ffff9d81984a0700 x1862023066300800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765543.093451:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066300800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765543.093457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765543.093459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765543.093461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765543.093464:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765543.093474:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765543.093479:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765543/real 1775765543] req@ffff9d81984a3b80 x1862023066300928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765543.093486:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066300928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765543.093491:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765543.093493:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765543.093494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066300928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765543.733003:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765544.116991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765544.757070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765545.140982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765545.781113:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765546.164969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765546.805111:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765547.189168:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765547.829111:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765548.213045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765548.213073:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765548.213084:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765548.213089:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765548.213102:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2379 00000100:00080000:0.0:1775765548.213114:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765548.213121:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765548.213164:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066301056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765548.213202:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765548.213206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765548.213207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765548.213215:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2379 00000100:00080000:0.0:1775765548.213218:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765548.213222:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765548.213243:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765548.213255:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066301184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765548.213267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765548.213271:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765548.213273:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765548.213287:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2379 00000100:00080000:0.0:1775765548.213291:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765548.213291:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066301056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765548.213300:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765548.213315:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765548.213376:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066301312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765548.213394:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765548.213400:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765548.213402:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765548.213409:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2379 00000100:00080000:0.0:1775765548.213440:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765548.213445:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765548.213453:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765548.213457:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765548.213486:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066301440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765548.213504:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765548.213511:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765548.213515:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765548.213523:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2379 00000100:00080000:0.0:1775765548.213530:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765548.213537:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765548.213546:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765548/real 1775765548] req@ffff9d81901ae680 x1862023066301056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775765548.213547:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765548.213568:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066301568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765548.213645:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066301056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765548.213661:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765548.213717:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765548.213726:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066301056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765548.213774:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765548.213828:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765548.213856:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765548/real 1775765548] req@ffff9d81901af100 x1862023066301184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765548.213877:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066301184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765548.213898:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765548.213938:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765548.213944:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765548.213953:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765548.213986:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765548.214002:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765548/real 1775765548] req@ffff9d81901ad180 x1862023066301312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765548.214018:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066301312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765548.214037:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765548.214042:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765548.214046:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765548.214072:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765548.214095:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765548.214106:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765548/real 1775765548] req@ffff9d81901afb80 x1862023066301440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765548.214121:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066301440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765548.214138:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765548.214145:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765548.214151:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765548.214162:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765548.214228:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765548.214250:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765548/real 1775765548] req@ffff9d81901aed80 x1862023066301568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765548.214275:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066301568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765548.214287:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765548.214291:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765548.214296:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765548.853128:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765549.237036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765549.877055:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765550.261078:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765550.901021:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765551.285036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765551.925133:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765552.308988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765552.949088:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765553.332998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765553.333198:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765553.333209:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765553.333214:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765553.333225:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2384 00000100:00080000:2.0:1775765553.333239:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765553.333247:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765553.333322:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066301696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765553.333359:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765553.333364:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765553.333367:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765553.333371:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2384 00000100:00080000:2.0:1775765553.333376:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765553.333383:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765553.333396:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765553.333415:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066301824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765553.333422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066301696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765553.333447:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765553.333451:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765553.333454:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765553.333461:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2384 00000100:00080000:2.0:1775765553.333465:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765553.333469:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765553.333475:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765553.333491:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066301952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765553.333502:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765553.333506:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765553.333509:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765553.333514:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2384 00000100:00080000:2.0:1775765553.333530:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765553.333534:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765553.333540:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765553.333552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765553.333558:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066302080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765553.333572:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765553.333577:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765553.333580:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765553.333586:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2384 00000100:00080000:2.0:1775765553.333591:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765553.333595:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765553.333601:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765553.333614:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066302208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765553.333656:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765553/real 1775765553] req@ffff9d81984a3480 x1862023066301696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765553.333697:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066301696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765553.333718:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765553.333725:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765553.333730:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066301696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765553.333761:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765553.333798:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765553.333815:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765553/real 1775765553] req@ffff9d81984a3b80 x1862023066301824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765553.333832:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066301824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765553.333843:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765553.333848:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765553.333852:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765553.333859:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765553.333919:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765553.333934:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765553/real 1775765553] req@ffff9d81984a0700 x1862023066301952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765553.333949:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066301952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765553.333960:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765553.333964:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765553.333968:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066301952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765553.333976:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765553.333998:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765553.334009:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765553/real 1775765553] req@ffff9d81984a2d80 x1862023066302080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765553.334023:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066302080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765553.334035:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765553.334038:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765553.334042:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765553.334049:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765553.334068:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765553.334078:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765553/real 1775765553] req@ffff9d81984a2a00 x1862023066302208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765553.334108:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066302208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765553.334116:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765553.334119:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765553.334122:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765553.973082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765554.357061:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765554.997110:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765555.381037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765556.021205:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765556.405066:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765557.045201:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765557.429079:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765558.069223:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765558.453959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765558.454025:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765558.454037:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765558.454042:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765558.454054:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2389 00000100:00080000:0.0:1775765558.454068:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765558.454077:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765558.454125:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066302336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765558.454182:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765558.454187:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765558.454190:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765558.454194:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2389 00000100:00080000:0.0:1775765558.454199:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765558.454220:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765558.454257:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765558.454277:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066302464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765558.454292:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765558.454296:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765558.454298:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765558.454306:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2389 00000100:00100000:3.0:1775765558.454306:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066302336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765558.454315:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765558.454320:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765558.454326:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765558.454347:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066302592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765558.454363:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765558.454369:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765558.454372:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765558.454378:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2389 00000800:00000100:3.0:1775765558.454385:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765558.454390:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765558.454400:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765558.454410:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765558.454433:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066302720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765558.454447:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765558/real 1775765558] req@ffff9d81901aed80 x1862023066302336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765558.454454:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765558.454469:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765558.454474:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765558.454484:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2389 00000100:00080000:0.0:1775765558.454490:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765558.454495:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765558.454503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066302336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775765558.454504:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765558.454523:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066302848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765558.454526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765558.454544:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765558.454551:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066302336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765558.454579:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765558.454607:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765558.454620:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765558/real 1775765558] req@ffff9d81901afb80 x1862023066302464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765558.454641:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066302464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765558.454649:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765558.454653:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765558.454656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765558.454662:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765558.454677:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765558.454703:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765558/real 1775765558] req@ffff9d81901ad180 x1862023066302592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765558.454716:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066302592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765558.454726:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765558.454729:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765558.454732:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765558.454738:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765558.454757:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765558.454767:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765558/real 1775765558] req@ffff9d81901af100 x1862023066302720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765558.454779:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066302720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765558.454797:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765558.454800:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765558.454804:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765558.454810:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765558.454825:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765558.454834:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765558/real 1775765558] req@ffff9d81901ae680 x1862023066302848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765558.454846:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066302848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765558.454856:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765558.454859:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765558.454862:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066302848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765559.093209:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765559.476995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765560.117145:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765560.500961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765561.141019:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765561.524942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765562.165022:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765562.549029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765563.189045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765563.572990:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765563.573016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765563.573023:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765563.573028:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765563.573038:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2395 00000100:00080000:0.0:1775765563.573048:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765563.573056:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765563.573109:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066302976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765563.573137:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765563.573140:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765563.573142:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765563.573145:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2395 00000100:00080000:0.0:1775765563.573148:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765563.573151:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765563.573159:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765563.573169:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066303104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765563.573180:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765563.573199:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765563.573201:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765563.573206:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2395 00000100:00080000:0.0:1775765563.573226:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765563.573242:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066302976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765563.573243:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765563.573253:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765563.573269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066303232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765563.573279:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765563.573281:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765563.573283:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765563.573285:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2395 00000100:00080000:0.0:1775765563.573297:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765563.573303:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765563.573303:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775765563.573311:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765563.573326:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066303360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765563.573336:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765563.573340:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765563.573343:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765563.573344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765563/real 1775765563] req@ffff9d81901ae680 x1862023066302976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765563.573352:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2395 00000100:00080000:0.0:1775765563.573360:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765563.573365:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765563.573369:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066302976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775765563.573370:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775765563.573382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775765563.573386:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066303488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765563.573389:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765563.573396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066302976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765563.573405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765563.573423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765563.573431:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765563/real 1775765563] req@ffff9d81901af100 x1862023066303104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765563.573439:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066303104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765563.573445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765563.573447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765563.573449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765563.573466:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765563.573477:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765563.573483:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765563/real 1775765563] req@ffff9d81901ad180 x1862023066303232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765563.573490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066303232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765563.573495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765563.573497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765563.573499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765563.573503:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765563.573513:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765563.573518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765563/real 1775765563] req@ffff9d81901afb80 x1862023066303360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765563.573525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066303360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765563.573531:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765563.573533:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765563.573535:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765563.573538:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765563.573547:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765563.573553:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765563/real 1775765563] req@ffff9d81901aed80 x1862023066303488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765563.573559:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066303488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765563.573565:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765563.573566:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765563.573568:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765564.213061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765564.596993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765565.237055:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765565.620965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765566.261127:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765566.644963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765567.285010:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765567.668953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765568.309057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765568.693008:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765568.693016:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1775765568.693018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765568.693020:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765568.693048:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2400 00000100:00080000:2.0:1775765568.693059:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765568.693067:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765568.693132:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066303616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765568.693200:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765568.693202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765568.693204:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765568.693206:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2400 00000100:00080000:2.0:1775765568.693209:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765568.693212:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765568.693220:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765568.693230:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066303744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765568.693238:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066303616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765568.693239:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765568.693246:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765568.693249:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765568.693253:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2400 00000100:00080000:2.0:1775765568.693255:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765568.693257:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765568.693260:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765568.693269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066303872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765568.693274:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765568.693277:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765568.693278:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765568.693281:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2400 00000100:00080000:2.0:1775765568.693292:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765568.693294:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765568.693297:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765568.693302:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765568.693308:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066304000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765568.693316:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765568.693318:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765568.693319:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765568.693321:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2400 00000100:00080000:2.0:1775765568.693324:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765568.693326:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765568.693329:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765568.693336:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066304128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765568.693350:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765568/real 1775765568] req@ffff9d81984a0000 x1862023066303616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765568.693367:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066303616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765568.693375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765568.693378:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765568.693381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066303616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765568.693408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765568.693423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765568.693430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765568/real 1775765568] req@ffff9d81984a2300 x1862023066303744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765568.693438:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066303744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765568.693444:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765568.693446:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765568.693448:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765568.693452:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765568.693473:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765568.693478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765568/real 1775765568] req@ffff9d81984a1f80 x1862023066303872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765568.693489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066303872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765568.693495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765568.693497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765568.693498:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066303872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765568.693501:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765568.693511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765568.693517:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765568/real 1775765568] req@ffff9d81984a1180 x1862023066304000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765568.693524:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066304000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765568.693540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765568.693541:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765568.693543:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765568.693546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765568.693556:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765568.693564:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765568/real 1775765568] req@ffff9d81984a1500 x1862023066304128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765568.693576:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066304128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765568.693591:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765568.693594:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765568.693598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765569.333204:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765569.716955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765570.357231:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765570.740965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765571.381094:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765571.764930:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765572.405077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765572.788926:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765573.429048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765573.812928:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765573.812947:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765573.812953:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765573.812957:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765573.812966:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2405 00000100:00080000:2.0:1775765573.812976:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765573.812982:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765573.813021:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066304256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765573.813058:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765573.813062:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765573.813064:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765573.813069:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2405 00000100:00080000:2.0:1775765573.813074:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765573.813080:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765573.813101:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765573.813102:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066304256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765573.813121:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066304384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765573.813132:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765573.813136:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765573.813138:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765573.813154:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2405 00000100:00080000:2.0:1775765573.813167:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765573.813172:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765573.813175:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765573.813182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765573.813197:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066304512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765573.813208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765573.813212:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765573.813214:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765573.813220:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2405 00000100:00100000:3.0:1775765573.813220:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765573/real 1775765573] req@ffff9d81984a2680 x1862023066304256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765573.813229:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765573.813236:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765573.813241:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765573.813246:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066304256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765573.813253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066304640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765573.813267:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765573.813268:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765573.813290:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765573.813294:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765573.813294:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775765573.813301:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2405 00000100:00080000:2.0:1775765573.813306:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765573.813306:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066304256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765573.813313:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765573.813318:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765573.813326:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066304768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765573.813340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765573.813365:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765573.813375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765573/real 1775765573] req@ffff9d81984a1880 x1862023066304384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765573.813384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066304384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765573.813390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765573.813392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765573.813394:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765573.813398:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765573.813412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765573.813423:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765573/real 1775765573] req@ffff9d81984a3800 x1862023066304512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765573.813435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066304512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765573.813443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765573.813447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765573.813451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765573.813457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765573.813476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765573.813484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765573/real 1775765573] req@ffff9d81984a3100 x1862023066304640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765573.813492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066304640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765573.813499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765573.813501:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765573.813503:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765573.813507:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765573.813518:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765573.813523:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765573/real 1775765573] req@ffff9d81984a0e00 x1862023066304768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765573.813530:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066304768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765573.813536:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765573.813538:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765573.813539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066304768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765574.453146:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765574.836966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765575.477021:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765575.860959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765576.501133:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765576.884978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765577.525157:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765577.908993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765578.549110:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765578.933031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765578.933091:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765578.933099:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765578.933103:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765578.933113:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2410 00000100:00080000:0.0:1775765578.933124:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765578.933130:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765578.933172:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066304896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765578.933205:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765578.933211:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765578.933213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765578.933220:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2410 00000100:00080000:0.0:1775765578.933225:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765578.933231:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765578.933255:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765578.933273:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066305024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765578.933273:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066304896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765578.933288:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765578.933293:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765578.933295:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765578.933300:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2410 00000100:00080000:0.0:1775765578.933305:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765578.933310:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765578.933316:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765578.933327:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066305152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765578.933339:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775765578.933341:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765578.933344:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765578.933346:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765578.933350:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2410 00000100:00080000:0.0:1775765578.933367:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765578.933372:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765578.933378:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765578.933391:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066305280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765578.933405:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765578.933409:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775765578.933410:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765578/real 1775765578] req@ffff9d81901aed80 x1862023066304896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765578.933412:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765578.933417:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2410 00000100:00080000:0.0:1775765578.933421:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765578.933425:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765578.933431:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765578.933435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066304896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765578.933442:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066305408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765578.933450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765578.933456:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765578.933464:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066304896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765578.933492:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765578.933524:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765578.933538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765578/real 1775765578] req@ffff9d81901afb80 x1862023066305024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765578.933551:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066305024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765578.933558:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765578.933561:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765578.933564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765578.933569:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765578.933586:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765578.933593:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765578/real 1775765578] req@ffff9d81901ad180 x1862023066305152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765578.933603:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066305152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765578.933610:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765578.933613:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765578.933616:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765578.933621:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765578.933634:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765578.933641:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765578/real 1775765578] req@ffff9d81901af100 x1862023066305280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765578.933651:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066305280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765578.933658:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765578.933661:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765578.933663:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765578.933680:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765578.933697:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765578.933705:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765578/real 1775765578] req@ffff9d81901ae680 x1862023066305408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765578.933715:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066305408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765578.933728:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765578.933731:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765578.933734:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765579.573076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765579.956950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765580.597190:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765580.981036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765581.621115:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765582.005185:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765582.645072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765583.028995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765583.669107:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765584.053065:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765584.053067:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765584.053089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765584.053099:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765584.053121:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2415 00000100:00080000:0.0:1775765584.053134:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765584.053143:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765584.053192:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066305536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765584.053246:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765584.053253:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765584.053256:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765584.053263:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2415 00000100:00080000:0.0:1775765584.053270:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765584.053277:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765584.053290:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765584.053308:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066305664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765584.053322:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066305536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765584.053327:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765584.053335:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765584.053338:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765584.053344:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2415 00000100:00080000:0.0:1775765584.053347:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765584.053351:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765584.053357:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765584.053373:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066305792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765584.053384:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765584.053388:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765584.053390:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765584.053394:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2415 00000100:00080000:0.0:1775765584.053398:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765584.053401:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765584.053406:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765584.053408:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765584.053426:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066305920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765584.053437:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765584.053441:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765584.053443:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765584.053449:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2415 00000100:00080000:0.0:1775765584.053453:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765584.053457:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765584.053463:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765584.053474:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765584/real 1775765584] req@ffff9d81901ae680 x1862023066305536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765584.053480:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066306048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765584.053536:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066305536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765584.053551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765584.053556:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765584.053560:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066305536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765584.053596:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765584.053620:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765584.053632:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765584/real 1775765584] req@ffff9d81901af100 x1862023066305664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765584.053661:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066305664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765584.053686:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765584.053692:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765584.053698:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765584.053709:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765584.053749:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765584.053770:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765584/real 1775765584] req@ffff9d81901ad180 x1862023066305792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765584.053808:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066305792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765584.053823:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765584.053827:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765584.053830:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765584.053837:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765584.053862:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765584.053873:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765584/real 1775765584] req@ffff9d81901afb80 x1862023066305920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765584.053905:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066305920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765584.053916:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765584.053984:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765584.054011:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066305920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765584.054023:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765584.054058:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765584.054077:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765584/real 1775765584] req@ffff9d81901aed80 x1862023066306048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765584.054112:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066306048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765584.054130:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765584.054134:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765584.054137:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765584.693096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765585.077040:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765585.717174:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765586.100946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765586.741200:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765587.124986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765587.765105:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765588.148978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765588.789196:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765589.173009:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765589.173010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765589.173019:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765589.173024:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765589.173034:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2420 00000100:00080000:0.0:1775765589.173043:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765589.173048:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765589.173083:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066306176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765589.173108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765589.173110:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765589.173112:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765589.173127:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2420 00000100:00080000:0.0:1775765589.173129:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765589.173131:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765589.173139:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765589.173146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066306304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765589.173158:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765589.173160:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765589.173162:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765589.173164:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2420 00000100:00080000:0.0:1775765589.173167:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765589.173169:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765589.173172:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765589.173180:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066306432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765589.173184:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066306176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765589.173189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765589.173204:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765589.173205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765589.173208:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2420 00000100:00080000:0.0:1775765589.173210:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765589.173212:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765589.173215:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765589.173223:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066306560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765589.173229:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765589.173231:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765589.173233:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765589.173235:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2420 00000100:00080000:0.0:1775765589.173252:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765589.173254:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765589.173257:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775765589.173264:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765589.173266:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066306688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765589.173322:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765589/real 1775765589] req@ffff9d81901aed80 x1862023066306176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765589.173343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066306176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765589.173355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765589.173359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765589.173363:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066306176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765589.173371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765589.173395:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765589.173419:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765589/real 1775765589] req@ffff9d81901afb80 x1862023066306304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765589.173433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066306304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765589.173441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765589.173444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765589.173447:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765589.173470:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765589.173490:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765589.173512:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765589/real 1775765589] req@ffff9d81901ad180 x1862023066306432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765589.173526:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066306432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765589.173534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765589.173537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765589.173540:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765589.173547:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765589.173567:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765589.173578:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765589/real 1775765589] req@ffff9d81901af100 x1862023066306560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765589.173590:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066306560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765589.173605:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765589.173609:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765589.173612:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765589.173618:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765589.173635:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765589.173645:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765589/real 1775765589] req@ffff9d81901ae680 x1862023066306688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765589.173657:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066306688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765589.173666:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765589.173669:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765589.173672:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765589.813054:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765590.196990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765590.837165:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765591.221020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765591.861102:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765592.245099:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765592.885149:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765593.268979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765593.909137:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765594.293018:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765594.293019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765594.293031:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765594.293037:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765594.293047:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2425 00000100:00080000:0.0:1775765594.293060:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765594.293068:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765594.293112:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066306816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765594.293144:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765594.293148:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765594.293150:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765594.293153:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2425 00000100:00080000:0.0:1775765594.293157:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765594.293160:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765594.293170:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765594.293182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066306944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765594.293204:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765594.293211:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765594.293214:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765594.293219:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066306816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765594.293224:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2425 00000100:00080000:0.0:1775765594.293232:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765594.293235:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765594.293240:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765594.293254:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066307072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765594.293290:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765594.293294:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765594.293296:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765594.293300:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2425 00000100:00080000:0.0:1775765594.293316:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765594.293320:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765594.293352:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765594.293365:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066307200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765594.293375:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765594.293378:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765594.293380:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765594.293385:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2425 00000100:00080000:0.0:1775765594.293390:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765594.293395:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765594.293403:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765594.293412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765594.293418:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066307328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765594.293476:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765594.293519:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765594.293551:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765594.933095:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765595.317063:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765595.957017:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765596.341026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765596.981026:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765597.365152:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775765597.365173:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775765597.365223:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775765597.365240:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765597.365282:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765597.365295:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765597.365311:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765597.365323:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765597.365337:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765594/real 1775765597] req@ffff9d81901ad180 x1862023066307072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765597.365371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066307072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765597.365386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765597.365392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765597.365397:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765597.365405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765594/real 1775765597] req@ffff9d81901afb80 x1862023066307200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765597.365422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066307200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765597.365435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765597.365439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765597.365443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765597.365451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765594/real 1775765597] req@ffff9d81901aed80 x1862023066307328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765597.365467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066307328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765597.365477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765597.365481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765597.365485:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307328:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765597.365493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765594/real 1775765597] req@ffff9d81901ae680 x1862023066306816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765597.365507:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066306816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765597.365516:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765597.365520:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765597.365524:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066306816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765597.365531:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765594/real 1775765597] req@ffff9d81901af100 x1862023066306944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765597.365544:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066306944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765597.365553:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765597.365558:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765597.365562:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066306944:192.168.202.105@tcp:38:kworker.0 00010000:00010000:3.0:1775765598.005063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765598.389079:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765599.029058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765599.412975:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765599.412985:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765599.412989:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765599.412999:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2430 00000100:00080000:0.0:1775765599.413012:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765599.413020:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765599.413039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:0.0:1775765599.413057:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066307456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765599.413095:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765599.413098:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765599.413100:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765599.413103:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2430 00000100:00080000:0.0:1775765599.413106:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765599.413109:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765599.413116:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765599.413126:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066307584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765599.413133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765599.413136:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765599.413137:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765599.413140:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2430 00000100:00080000:0.0:1775765599.413144:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765599.413146:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066307456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765599.413150:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765599.413158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765599.413167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066307712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765599.413179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765599.413183:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765599.413184:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765599.413189:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2430 00000100:00080000:0.0:1775765599.413203:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765599.413208:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765599.413212:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765599.413229:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765599.413244:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066307840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765599.413251:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765599.413254:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765599.413255:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765599.413259:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2430 00000100:00080000:0.0:1775765599.413268:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765599.413273:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765599/real 1775765599] req@ffff9d81901af100 x1862023066307456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765599.413274:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765599.413279:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765599.413293:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066307968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765599.413304:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066307456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765599.413326:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765599.413332:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765599.413337:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066307456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765599.413346:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765599.413370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765599.413383:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765599/real 1775765599] req@ffff9d81901ae680 x1862023066307584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765599.413397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066307584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765599.413406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765599.413409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765599.413412:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765599.413436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765599.413458:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765599.413469:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765599/real 1775765599] req@ffff9d81901aed80 x1862023066307712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765599.413504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066307712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765599.413520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765599.413524:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765599.413528:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765599.413534:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765599.413555:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765599.413568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765599/real 1775765599] req@ffff9d81901afb80 x1862023066307840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765599.413580:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066307840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765599.413590:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765599.413594:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765599.413597:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765599.413603:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765599.413623:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765599.413633:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765599/real 1775765599] req@ffff9d81901ad180 x1862023066307968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765599.413643:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066307968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765599.413652:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765599.413655:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765599.413658:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066307968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765600.053131:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765600.437083:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765601.077014:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765601.460985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765602.101059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765602.484997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765603.125060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765603.508942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765604.149218:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765604.533009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765604.533044:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765604.533048:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765604.533052:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765604.533060:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2436 00000100:00080000:2.0:1775765604.533071:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765604.533078:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765604.533121:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066308096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765604.533140:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765604.533143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765604.533144:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765604.533146:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2436 00000100:00080000:2.0:1775765604.533148:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765604.533152:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765604.533159:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765604.533169:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066308224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765604.533175:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765604.533188:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765604.533189:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765604.533192:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2436 00000100:00080000:2.0:1775765604.533194:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765604.533198:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066308096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765604.533199:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765604.533205:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765604.533212:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066308352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765604.533219:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765604.533221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765604.533222:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765604.533225:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2436 00000100:00080000:2.0:1775765604.533227:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765604.533229:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765604.533232:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765604.533241:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066308480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765604.533247:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765604.533249:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765604.533251:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765604.533253:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2436 00000100:00080000:2.0:1775765604.533257:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765604.533260:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765604.533262:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775765604.533267:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765604.533271:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066308608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765604.533324:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765604/real 1775765604] req@ffff9d81984a0a80 x1862023066308096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765604.533339:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066308096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765604.533347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765604.533351:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765604.533355:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066308096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765604.533360:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765604.533376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765604.533394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765604/real 1775765604] req@ffff9d81984a0e00 x1862023066308224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765604.533402:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066308224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765604.533408:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765604.533410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765604.533413:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765604.533429:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765604.533440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765604.533446:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765604/real 1775765604] req@ffff9d81984a3100 x1862023066308352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765604.533454:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066308352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765604.533459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765604.533460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765604.533462:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765604.533466:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765604.533476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765604.533482:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765604/real 1775765604] req@ffff9d81984a3800 x1862023066308480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765604.533489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066308480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765604.533495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765604.533497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765604.533499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765604.533502:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765604.533511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765604.533524:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765604/real 1775765604] req@ffff9d81984a1880 x1862023066308608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765604.533532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066308608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765604.533537:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765604.533539:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765604.533540:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765605.173007:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765605.557062:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765606.197089:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765606.580970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765607.221077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765607.605019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765608.245073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765608.629026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765609.269137:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765609.653023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765609.653066:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765609.653079:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765609.653085:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765609.653110:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2441 00000100:00080000:0.0:1775765609.653127:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765609.653137:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765609.653212:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066308736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765609.653264:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765609.653268:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765609.653271:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765609.653276:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2441 00000100:00080000:0.0:1775765609.653282:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765609.653289:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765609.653301:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765609.653330:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066308736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765609.653332:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066308864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765609.653360:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765609.653367:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765609.653371:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765609.653380:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2441 00000100:00080000:0.0:1775765609.653402:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765609.653413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765609.653416:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765609.653431:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765609.653455:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066308992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765609.653467:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765609.653472:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765609.653475:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765609.653480:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2441 00000100:00080000:0.0:1775765609.653486:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765609.653488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765609/real 1775765609] req@ffff9d81901ad180 x1862023066308736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765609.653496:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765609.653512:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765609.653531:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066308736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765609.653535:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066309120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765609.653584:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765609.653585:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765609.653596:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765609.653600:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765609.653601:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:0.0:1775765609.653617:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2441 00000100:00100000:3.0:1775765609.653619:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066308736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765609.653628:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765609.653636:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765609.653642:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765609.653658:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765609.653660:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066309248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765609.653711:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765609.653730:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765609/real 1775765609] req@ffff9d81901afb80 x1862023066308864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765609.653786:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066308864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765609.653797:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765609.653801:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765609.653806:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765609.653813:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765609.653835:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765609.653847:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765609/real 1775765609] req@ffff9d81901aed80 x1862023066308992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765609.653860:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066308992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765609.653869:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765609.653873:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765609.653876:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066308992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765609.653917:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765609.653939:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765609.653951:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765609/real 1775765609] req@ffff9d81901ae680 x1862023066309120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765609.653965:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066309120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765609.653976:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765609.653980:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765609.653984:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765609.654009:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765609.654027:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765609.654037:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765609/real 1775765609] req@ffff9d81901af100 x1862023066309248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765609.654063:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066309248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765609.654074:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765609.654078:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765609.654081:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765610.293087:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765610.676988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765611.317186:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765611.700991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765612.341085:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765612.724969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765613.365058:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765613.749052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765614.389045:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765614.772974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765614.773023:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765614.773033:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765614.773040:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765614.773052:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2446 00000100:00080000:0.0:1775765614.773065:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765614.773074:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765614.773124:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066309376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765614.773182:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765614.773185:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765614.773187:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765614.773191:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2446 00000100:00080000:0.0:1775765614.773195:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765614.773198:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765614.773207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765614.773220:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066309504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765614.773230:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765614.773235:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765614.773236:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765614.773241:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2446 00000100:00080000:0.0:1775765614.773244:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765614.773246:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765614.773252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765614.773256:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066309376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765614.773265:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066309632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765614.773298:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765614.773301:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765614.773302:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765614.773305:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2446 00000100:00080000:0.0:1775765614.773309:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765614.773312:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765614.773317:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765614.773327:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066309760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765614.773335:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765614.773338:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765614.773342:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775765614.773343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765614.773348:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2446 00000100:00080000:0.0:1775765614.773352:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765614.773356:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765614.773360:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765614.773379:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066309888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765614.773420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765614/real 1775765614] req@ffff9d81901af100 x1862023066309376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765614.773448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066309376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765614.773463:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765614.773467:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765614.773472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066309376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765614.773480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765614.773506:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765614.773518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765614/real 1775765614] req@ffff9d81901ae680 x1862023066309504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765614.773529:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066309504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765614.773536:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765614.773539:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765614.773542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765614.773562:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765614.773577:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765614.773584:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765614/real 1775765614] req@ffff9d81901aed80 x1862023066309632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765614.773594:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066309632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765614.773601:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765614.773603:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765614.773606:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765614.773611:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765614.773624:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765614.773631:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765614/real 1775765614] req@ffff9d81901afb80 x1862023066309760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765614.773640:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066309760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765614.773648:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765614.773650:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765614.773653:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765614.773658:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765614.773670:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765614.773677:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765614/real 1775765614] req@ffff9d81901ad180 x1862023066309888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765614.773686:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066309888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765614.773713:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765614.773718:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765614.773722:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066309888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765615.413031:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765615.797042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765616.437032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765616.820941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765617.461073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765617.845005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765618.485173:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765618.869000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765619.509128:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765619.892990:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765619.892994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765619.893006:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765619.893012:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765619.893026:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2451 00000100:00080000:0.0:1775765619.893041:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765619.893061:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765619.893108:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066310016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765619.893142:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765619.893146:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765619.893148:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765619.893152:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2451 00000100:00080000:0.0:1775765619.893156:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765619.893160:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765619.893170:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765619.893184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066310144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765619.893195:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765619.893199:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765619.893202:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765619.893206:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2451 00000100:00080000:0.0:1775765619.893209:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765619.893211:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765619.893217:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765619.893229:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066310272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765619.893242:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765619.893246:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066310016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765619.893248:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765619.893254:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765619.893269:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2451 00000100:00080000:0.0:1775765619.893272:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765619.893275:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765619.893280:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765619.893292:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066310400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765619.893301:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765619.893303:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765619.893305:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765619.893311:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2451 00000100:00080000:0.0:1775765619.893314:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765619.893317:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765619.893321:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765619.893330:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066310528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765619.893375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765619.893474:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765619/real 1775765619] req@ffff9d81901ad180 x1862023066310016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765619.893504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066310016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765619.893519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765619.893525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765619.893530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066310016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765619.893553:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765619.893605:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765619.893622:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765619/real 1775765619] req@ffff9d81901afb80 x1862023066310144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765619.893638:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066310144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765619.893771:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765619.893777:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765619.893781:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765619.893818:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765619.893922:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765619.893958:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765619/real 1775765619] req@ffff9d81901aed80 x1862023066310272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765619.893975:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066310272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765619.893987:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765619.893992:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765619.893996:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765619.894004:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765619.894050:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765619.894067:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765619/real 1775765619] req@ffff9d81901ae680 x1862023066310400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765619.894086:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066310400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765619.894113:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765619.894119:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765619.894124:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765619.894136:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765619.894176:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765619.894189:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765619/real 1775765619] req@ffff9d81901af100 x1862023066310528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765619.894209:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066310528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765619.894219:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765619.894223:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765619.894226:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765620.533068:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765620.916986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765621.557033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765621.940970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765622.581028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765622.964949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765623.605154:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765623.989013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765624.629091:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765625.013005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765625.013076:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765625.013091:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765625.013098:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765625.013115:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2456 00000100:00080000:0.0:1775765625.013136:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765625.013152:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765625.013233:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066310656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765625.013289:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765625.013297:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765625.013300:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765625.013308:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2456 00000100:00080000:0.0:1775765625.013315:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765625.013323:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765625.013340:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765625.013366:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066310784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765625.013407:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765625.013419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066310656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765625.013420:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765625.013427:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765625.013449:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2456 00000100:00080000:0.0:1775765625.013457:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765625.013466:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765625.013479:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765625.013511:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066310912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765625.013532:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765625.013540:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765625.013548:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775765625.013551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765625.013571:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2456 00000100:00080000:0.0:1775765625.013581:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765625.013590:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765625.013632:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765625.013662:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066311040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765625.013675:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765625/real 1775765625] req@ffff9d81901af100 x1862023066310656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765625.013729:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765625.013737:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775765625.013743:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066310656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765625.013744:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765625.013769:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2456 00000100:00080000:0.0:1775765625.013787:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765625.013812:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765625.013814:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775765625.013833:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:0.0:1775765625.013842:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765625.013842:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066310656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765625.013923:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765625.013956:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066311168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765625.013968:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765625.013993:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765625/real 1775765625] req@ffff9d81901ae680 x1862023066310784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765625.014016:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066310784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765625.014027:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765625.014036:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765625.014040:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765625.014050:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765625.014071:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765625.014083:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765625/real 1775765625] req@ffff9d81901aed80 x1862023066310912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765625.014097:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066310912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765625.014106:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765625.014110:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765625.014113:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066310912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765625.014120:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765625.014139:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765625.014149:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765625/real 1775765625] req@ffff9d81901afb80 x1862023066311040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765625.014162:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066311040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765625.014180:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765625.014184:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765625.014188:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765625.014206:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765625.014223:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765625.014232:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765625/real 1775765625] req@ffff9d81901ad180 x1862023066311168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765625.014245:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066311168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765625.014255:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765625.014258:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765625.014262:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765625.653118:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765626.037013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765626.677050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765627.060966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765627.701166:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765628.084978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765628.725136:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765629.108987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765629.749128:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765630.132991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765630.133034:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765630.133041:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765630.133046:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765630.133056:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2461 00000100:00080000:0.0:1775765630.133066:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765630.133098:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765630.133136:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066311296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765630.133157:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765630.133162:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765630.133164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765630.133166:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2461 00000100:00080000:0.0:1775765630.133169:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765630.133172:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765630.133179:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765630.133189:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066311424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765630.133197:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765630.133200:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765630.133201:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765630.133204:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2461 00000100:00080000:0.0:1775765630.133206:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765630.133224:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765630.133228:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765630.133230:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066311296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765630.133237:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066311552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765630.133244:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765630.133247:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765630.133249:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765630.133251:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2461 00000100:00080000:0.0:1775765630.133254:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765630.133257:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765630.133261:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765630.133269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066311680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765630.133276:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765630.133278:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765630.133279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765630.133281:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2461 00000100:00080000:0.0:1775765630.133284:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765630.133288:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765630.133290:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765630.133295:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765630.133303:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066311808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765630.133332:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765630/real 1775765630] req@ffff9d81901ad180 x1862023066311296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765630.133347:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066311296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765630.133355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765630.133359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765630.133362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066311296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765630.133366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765630.133380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765630.133388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765630/real 1775765630] req@ffff9d81901afb80 x1862023066311424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765630.133396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066311424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765630.133401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765630.133403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765630.133405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765630.133422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765630.133433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765630.133438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765630/real 1775765630] req@ffff9d81901aed80 x1862023066311552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765630.133446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066311552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765630.133451:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765630.133453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765630.133456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765630.133460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765630.133470:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765630.133475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765630/real 1775765630] req@ffff9d81901ae680 x1862023066311680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765630.133483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066311680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765630.133489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765630.133490:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765630.133492:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765630.133495:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765630.133505:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765630.133511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765630/real 1775765630] req@ffff9d81901af100 x1862023066311808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765630.133518:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066311808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765630.133523:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765630.133525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765630.133527:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066311808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765630.773071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765631.156971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765631.797101:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765632.181044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765632.821082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765633.205004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765633.845144:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765634.228986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765634.868995:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765635.252997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765635.252999:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765635.253013:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765635.253020:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765635.253034:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2466 00000100:00080000:0.0:1775765635.253049:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765635.253059:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765635.253123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066311936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765635.253166:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765635.253172:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765635.253176:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765635.253182:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2466 00000100:00080000:0.0:1775765635.253190:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765635.253197:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765635.253211:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765635.253227:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066311936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765635.253236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066312064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765635.253258:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765635.253264:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765635.253267:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765635.253275:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2466 00000100:00080000:0.0:1775765635.253302:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765635.253310:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765635.253316:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775765635.253323:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765635.253347:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066312192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765635.253363:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765635.253369:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765635.253372:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765635.253381:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2466 00000100:00100000:3.0:1775765635.253384:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765635/real 1775765635] req@ffff9d81901af100 x1862023066311936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765635.253390:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765635.253405:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765635.253416:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765635.253422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066311936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765635.253440:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066312320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765635.253446:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765635.253460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765635.253470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066311936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765635.253492:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765635.253520:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775765635.253526:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312064:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775765635.253527:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765635.253543:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2466 00000100:00080000:0.0:1775765635.253551:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765635.253557:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765635.253566:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775765635.253580:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765635.253591:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066312448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765635.253606:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765635/real 1775765635] req@ffff9d81901ae680 x1862023066312064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765635.253625:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066312064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765635.253635:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765635.253640:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765635.253644:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765635.253651:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765635.253676:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765635.253686:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765635/real 1775765635] req@ffff9d81901aed80 x1862023066312192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765635.253698:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066312192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765635.253705:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765635.253708:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765635.253711:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765635.253717:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765635.253733:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765635.253741:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765635/real 1775765635] req@ffff9d81901afb80 x1862023066312320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765635.253753:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066312320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765635.253762:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765635.253765:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765635.253767:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765635.253783:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765635.253798:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765635.253809:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765635/real 1775765635] req@ffff9d81901ad180 x1862023066312448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765635.253840:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066312448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765635.253854:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765635.253859:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765635.253866:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765635.893228:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765636.276947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765636.917051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765637.300964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765637.941097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765638.325007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765638.965017:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765639.348966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765639.989053:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765640.372980:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765640.372987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765640.372999:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765640.373004:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765640.373017:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2471 00000100:00080000:0.0:1775765640.373031:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765640.373039:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765640.373089:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066312576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765640.373123:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765640.373127:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765640.373129:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765640.373134:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2471 00000100:00080000:0.0:1775765640.373138:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765640.373143:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765640.373156:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765640.373165:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066312576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765640.373172:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066312704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765640.373184:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765640.373189:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765640.373191:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765640.373196:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2471 00000100:00080000:0.0:1775765640.373200:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765640.373205:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765640.373211:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765640.373225:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066312832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765640.373236:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765640.373251:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765640.373255:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765640.373257:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765640.373263:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2471 00000100:00080000:0.0:1775765640.373268:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765640.373273:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765640.373279:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765640.373295:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066312960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765640.373296:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765640/real 1775765640] req@ffff9d81901ad180 x1862023066312576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765640.373327:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765640.373331:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765640.373335:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765640.373335:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066312576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765640.373347:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2471 00000100:00080000:3.0:1775765640.373350:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765640.373352:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765640.373356:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:0.0:1775765640.373362:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765640.373362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066312576:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775765640.373373:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765640.373388:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066313088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765640.373397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765640.373427:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765640.373442:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765640/real 1775765640] req@ffff9d81901afb80 x1862023066312704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765640.373473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066312704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765640.373483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765640.373486:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765640.373490:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765640.373497:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765640.373520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765640.373533:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765640/real 1775765640] req@ffff9d81901aed80 x1862023066312832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765640.373545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066312832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765640.373553:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765640.373556:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765640.373559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765640.373564:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765640.373583:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765640.373593:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765640/real 1775765640] req@ffff9d81901ae680 x1862023066312960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765640.373603:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066312960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765640.373614:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765640.373618:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765640.373621:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066312960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765640.373638:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765640.373657:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765640.373665:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765640/real 1775765640] req@ffff9d81901af100 x1862023066313088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765640.373673:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066313088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765640.373680:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765640.373683:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765640.373685:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765641.013099:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765641.397006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765642.037154:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765642.420971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765643.061237:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765643.445014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765644.085031:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765644.468973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765645.109082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765645.492967:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765645.492982:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765645.492988:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765645.493002:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2477 00000100:00100000:2.0:1775765645.493012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765645.493018:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765645.493030:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765645.493081:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066313216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765645.493116:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765645.493120:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765645.493122:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765645.493126:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2477 00000100:00080000:3.0:1775765645.493130:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765645.493135:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765645.493147:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765645.493162:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066313344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765645.493167:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066313216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765645.493181:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765645.493185:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765645.493188:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765645.493195:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2477 00000100:00080000:3.0:1775765645.493199:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765645.493204:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765645.493211:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765645.493226:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066313472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765645.493237:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765645.493241:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765645.493243:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765645.493249:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2477 00000100:00080000:3.0:1775765645.493254:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775765645.493258:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765645.493263:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765645.493270:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765645.493301:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066313600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765645.493313:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765645.493316:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765645.493319:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765645.493324:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2477 00000100:00100000:0.0:1775765645.493329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765645/real 1775765645] req@ffff9d8188740a80 x1862023066313216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765645.493331:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765645.493341:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765645.493351:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765645.493362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066313216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765645.493368:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066313728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765645.493386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765645.493394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765645.493430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066313216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765645.493460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765645.493486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765645.493500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765645/real 1775765645] req@ffff9d8188741500 x1862023066313344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765645.493512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066313344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765645.493527:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765645.493531:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765645.493534:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765645.493541:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765645.493562:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765645.493573:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765645/real 1775765645] req@ffff9d8188741180 x1862023066313472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765645.493585:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066313472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765645.493594:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765645.493598:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765645.493602:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765645.493608:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765645.493629:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765645.493638:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765645/real 1775765645] req@ffff9d8188743100 x1862023066313600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765645.493646:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066313600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765645.493653:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765645.493656:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765645.493659:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765645.493663:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765645.493675:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765645.493680:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765645/real 1775765645] req@ffff9d8188740e00 x1862023066313728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765645.493687:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066313728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765645.493693:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765645.493695:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765645.493697:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765646.133232:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765646.517018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765647.157104:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765647.541042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765648.181049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765648.564997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765649.205123:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765649.589015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765650.229201:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765650.612945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765650.612947:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765650.612955:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765650.612960:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765650.612971:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2482 00000100:00080000:3.0:1775765650.612983:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765650.612991:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765650.613034:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066313856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765650.613064:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765650.613067:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765650.613069:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765650.613073:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2482 00000100:00080000:3.0:1775765650.613076:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765650.613079:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765650.613088:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765650.613098:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066313984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765650.613106:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765650.613109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765650.613111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765650.613115:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2482 00000100:00080000:3.0:1775765650.613117:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765650.613120:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765650.613123:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765650.613132:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066314112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765650.613141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765650.613144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765650.613146:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775765650.613152:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066313856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765650.613154:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2482 00000100:00080000:3.0:1775765650.613161:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765650.613165:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765650.613168:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765650.613182:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066314240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765650.613189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765650.613192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765650.613194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765650.613197:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2482 00000100:00080000:3.0:1775765650.613226:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765650.613228:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765650.613233:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775765650.613238:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765650.613248:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066314368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765650.613308:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765650/real 1775765650] req@ffff9d8188743b80 x1862023066313856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765650.613327:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066313856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765650.613338:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765650.613342:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765650.613347:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066313856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765650.613384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765650.613409:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765650.613421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765650/real 1775765650] req@ffff9d8188742d80 x1862023066313984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765650.613432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066313984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765650.613439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765650.613442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765650.613445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066313984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765650.613465:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765650.613479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765650.613487:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765650/real 1775765650] req@ffff9d8188743480 x1862023066314112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765650.613496:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066314112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765650.613503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765650.613505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765650.613507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765650.613512:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765650.613525:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765650.613532:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765650/real 1775765650] req@ffff9d8188740e00 x1862023066314240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765650.613547:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066314240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765650.613555:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765650.613557:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765650.613559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765650.613563:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765650.613575:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765650.613582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765650/real 1775765650] req@ffff9d8188743100 x1862023066314368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765650.613591:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066314368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765650.613599:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765650.613601:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765650.613605:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765651.253126:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765651.636994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765652.277069:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765652.660970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765653.301148:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765653.684968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765654.325144:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765654.709073:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765655.349163:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775765655.733007:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765655.733010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765655.733022:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765655.733027:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765655.733040:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2487 00000100:00080000:1.0:1775765655.733054:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765655.733064:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765655.733108:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066314496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765655.733158:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765655.733164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765655.733166:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765655.733172:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2487 00000100:00080000:1.0:1775765655.733178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765655.733183:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765655.733195:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765655.733214:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066314624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765655.733232:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765655.733233:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066314496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765655.733247:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765655.733249:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765655.733270:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2487 00000100:00080000:1.0:1775765655.733293:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765655.733301:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775765655.733338:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775765655.733344:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765655.733366:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066314752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765655.733381:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765655.733385:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765655.733388:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775765655.733392:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765655/real 1775765655] req@ffff9d8190551500 x1862023066314496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765655.733398:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2487 00000100:00080000:1.0:1775765655.733409:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765655.733416:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765655.733422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066314496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775765655.733426:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775765655.733438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765655.733443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765655.733448:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066314496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765655.733453:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066314880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765655.733469:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765655.733475:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314624:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775765655.733477:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765655.733483:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765655.733489:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2487 00000100:00080000:1.0:1775765655.733495:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765655.733499:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775765655.733507:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775765655.733507:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765655.733522:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765655/real 1775765655] req@ffff9d8190553480 x1862023066314624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775765655.733530:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066315008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765655.733540:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066314624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765655.733557:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765655.733562:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765655.733566:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765655.733571:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765655.733589:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765655.733596:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765655/real 1775765655] req@ffff9d8190553100 x1862023066314752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765655.733622:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066314752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765655.733629:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765655.733632:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765655.733635:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765655.733640:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765655.733654:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765655.733662:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765655/real 1775765655] req@ffff9d8190550e00 x1862023066314880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765655.733687:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066314880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765655.733694:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765655.733696:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765655.733698:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066314880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765655.733710:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765655.733734:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765655.733743:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765655/real 1775765655] req@ffff9d8190551880 x1862023066315008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765655.733752:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066315008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765655.733759:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765655.733761:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765655.733764:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765656.373146:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765656.757030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765657.397151:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765657.781000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765658.421081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765658.805119:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765659.445096:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765659.829006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765660.469036:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765660.853051:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765660.853065:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765660.853074:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765660.853080:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765660.853092:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2492 00000100:00080000:2.0:1775765660.853104:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765660.853114:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765660.853158:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066315136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765660.853191:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765660.853196:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765660.853198:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765660.853202:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2492 00000100:00080000:2.0:1775765660.853206:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765660.853211:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765660.853221:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765660.853237:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066315264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765660.853247:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066315136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765660.853276:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765660.853287:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765660.853290:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765660.853296:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2492 00000100:00080000:2.0:1775765660.853308:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765660.853312:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765660.853319:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765660.853336:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066315392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765660.853347:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765660.853351:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765660.853355:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765660.853358:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2492 00000100:00080000:2.0:1775765660.853363:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765660.853367:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765660.853373:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765660.853384:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066315520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765660.853394:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765660.853398:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765660.853400:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765660.853405:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2492 00000100:00080000:2.0:1775765660.853408:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765660.853413:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765660.853420:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765660.853433:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066315648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765660.853444:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765660.853490:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765660.853511:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765660.853555:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765661.493134:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765661.877186:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765662.517136:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765662.900995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765663.541090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765663.925165:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775765663.925342:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775765663.925402:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775765663.925433:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765663.925502:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765663.925514:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765663.925523:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765663.925533:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775765663.925578:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765660/real 1775765663] req@ffff9d81984a2300 x1862023066315648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765663.925631:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066315648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765663.925645:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765663.925651:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765663.925656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315648:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765663.925665:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765660/real 1775765663] req@ffff9d81984a1500 x1862023066315264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765663.925697:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066315264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765663.925706:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765663.925711:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765663.925715:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765663.925723:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765660/real 1775765663] req@ffff9d81984a1180 x1862023066315392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765663.925738:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066315392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765663.925747:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765663.925751:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765663.925755:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765663.925763:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765660/real 1775765663] req@ffff9d81984a1f80 x1862023066315520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765663.925778:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066315520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765663.925789:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765663.925793:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765663.925796:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765663.925804:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765660/real 1775765663] req@ffff9d81984a2680 x1862023066315136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765663.925817:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066315136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765663.925826:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765663.925831:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765663.925835:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066315136:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775765664.565186:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765664.949019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765665.589183:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765665.973087:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765665.973101:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765665.973101:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765665.973109:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765665.973122:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2497 00000100:00080000:3.0:1775765665.973136:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765665.973147:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765665.973227:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066315776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765665.973267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765665.973273:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765665.973275:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765665.973280:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2497 00000100:00080000:3.0:1775765665.973285:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765665.973291:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765665.973305:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765665.973330:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066315904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765665.973352:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765665.973360:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765665.973364:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765665.973373:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2497 00000100:00080000:3.0:1775765665.973392:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765665.973402:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765665.973413:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765665.973423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066315776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765665.973442:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066316032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765665.973462:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765665.973466:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765665.973469:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765665.973476:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2497 00000100:00080000:3.0:1775765665.973482:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765665.973487:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765665.973495:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765665.973513:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066316160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765665.973528:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765665.973532:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765665.973534:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765665.973538:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2497 00000100:00080000:3.0:1775765665.973554:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765665.973559:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765665.973565:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775765665.973577:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765665.973585:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066316288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765665.973691:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765665/real 1775765665] req@ffff9d8188741180 x1862023066315776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765665.973729:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066315776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765665.973749:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765665.973758:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765665.973765:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066315776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765665.973779:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765665.973822:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765665.973844:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765665/real 1775765665] req@ffff9d8188741500 x1862023066315904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765665.973865:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066315904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765665.973955:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765665.973962:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765665.973969:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066315904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765665.974040:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765665.974081:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765665.974102:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765665/real 1775765665] req@ffff9d8188740a80 x1862023066316032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765665.974134:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066316032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765665.974148:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765665.974153:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765665.974158:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765665.974183:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765665.974249:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765665.974268:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765665/real 1775765665] req@ffff9d8188741880 x1862023066316160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765665.974289:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066316160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765665.974317:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765665.974323:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765665.974328:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765665.974339:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765665.974376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765665.974394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765665/real 1775765665] req@ffff9d8188741f80 x1862023066316288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765665.974427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066316288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765665.974445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765665.974452:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765665.974458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765666.613180:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765666.997080:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765667.637107:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765668.020929:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765668.661155:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765669.045007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765669.685165:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765670.068966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765670.709138:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765671.092974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765671.093040:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765671.093049:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765671.093056:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765671.093073:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2502 00000100:00080000:0.0:1775765671.093093:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765671.093105:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765671.093223:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066316416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765671.093265:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765671.093270:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765671.093273:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765671.093278:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2502 00000100:00080000:0.0:1775765671.093284:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765671.093290:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765671.093304:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765671.093316:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066316416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765671.093327:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066316544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765671.093342:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765671.093346:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765671.093348:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765671.093353:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2502 00000100:00080000:0.0:1775765671.093357:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765671.093361:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765671.093366:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765671.093381:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066316672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765671.093392:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765671.093396:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765671.093398:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765671.093402:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2502 00000100:00080000:0.0:1775765671.093435:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765671.093440:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765671.093447:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765671.093457:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765671.093466:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066316800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765671.093480:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765671.093485:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765671.093487:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765671.093493:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2502 00000100:00080000:0.0:1775765671.093497:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765671.093501:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765671.093507:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765671.093524:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066316928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765671.093544:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765671/real 1775765671] req@ffff9d81901af100 x1862023066316416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765671.093574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066316416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765671.093589:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765671.093595:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765671.093600:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066316416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765671.093641:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765671.093690:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765671.093717:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765671/real 1775765671] req@ffff9d81901ae680 x1862023066316544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765671.093744:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066316544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765671.093774:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765671.093779:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765671.093783:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765671.093791:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765671.093819:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765671.093833:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765671/real 1775765671] req@ffff9d81901aed80 x1862023066316672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765671.093851:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066316672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765671.093878:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765671.093884:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765671.093890:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765671.093901:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765671.093978:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765671.093994:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765671/real 1775765671] req@ffff9d81901afb80 x1862023066316800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765671.094009:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066316800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765671.094021:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765671.094025:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765671.094029:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765671.094036:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765671.094056:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765671.094066:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765671/real 1775765671] req@ffff9d81901ad180 x1862023066316928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765671.094079:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066316928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765671.094089:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765671.094092:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765671.094096:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066316928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765671.733099:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765672.116990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765672.757176:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765673.140998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765673.781117:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765674.165055:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765674.805174:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765675.188983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765675.829198:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765676.212949:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765676.212959:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765676.212965:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765676.212975:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2507 00000100:00080000:0.0:1775765676.212987:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765676.212991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765676.212999:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765676.213046:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066317056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765676.213085:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765676.213098:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765676.213101:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765676.213125:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2507 00000100:00080000:0.0:1775765676.213130:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765676.213135:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765676.213147:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765676.213171:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066317184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765676.213174:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066317056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765676.213200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765676.213206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765676.213209:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765676.213218:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2507 00000100:00080000:0.0:1775765676.213224:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765676.213231:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765676.213241:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775765676.213263:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765676.213266:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066317312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765676.213284:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765676.213290:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765676.213293:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765676.213300:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2507 00000100:00080000:0.0:1775765676.213307:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765676.213314:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765676/real 1775765676] req@ffff9d81901ad180 x1862023066317056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765676.213318:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765676.213332:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765676.213343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066317056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765676.213355:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066317440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765676.213355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765676.213369:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:0.0:1775765676.213381:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765676.213381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066317056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765676.213392:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765676.213395:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765676.213401:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2507 00000100:00080000:0.0:1775765676.213405:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765676.213409:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765676.213415:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317184:192.168.202.105@tcp:38:kworker.0 00000008:00100000:0.0:1775765676.213417:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765676.213433:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066317568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765676.213446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765676.213461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765676/real 1775765676] req@ffff9d81901afb80 x1862023066317184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765676.213475:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066317184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765676.213483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765676.213486:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765676.213490:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765676.213495:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765676.213512:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765676.213520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765676/real 1775765676] req@ffff9d81901aed80 x1862023066317312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765676.213530:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066317312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765676.213537:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765676.213540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765676.213542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765676.213548:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765676.213562:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765676.213570:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765676/real 1775765676] req@ffff9d81901ae680 x1862023066317440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765676.213580:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066317440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765676.213588:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765676.213591:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765676.213593:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765676.213605:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765676.213618:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765676.213642:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765676/real 1775765676] req@ffff9d81901af100 x1862023066317568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765676.213661:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066317568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765676.213670:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765676.213673:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765676.213676:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765676.853173:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765677.236975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765677.877057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765678.260991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765678.901175:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765679.285020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765679.925122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765680.309044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765680.949056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765681.332992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765681.333006:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765681.333014:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765681.333019:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765681.333030:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2512 00000100:00080000:0.0:1775765681.333042:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765681.333050:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765681.333120:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066317696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765681.333153:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765681.333157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765681.333159:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765681.333163:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2512 00000100:00080000:0.0:1775765681.333168:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765681.333171:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765681.333181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765681.333195:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066317824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765681.333211:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765681.333217:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765681.333220:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765681.333225:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066317696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765681.333231:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2512 00000100:00080000:0.0:1775765681.333239:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765681.333245:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765681.333255:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765681.333276:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066317952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765681.333288:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765681.333292:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765681.333294:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765681.333301:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2512 00000800:00000100:3.0:1775765681.333301:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765681.333310:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765681.333316:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765681.333323:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765681.333337:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066318080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765681.333346:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765681.333350:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765681.333352:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765681.333358:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2512 00000100:00100000:3.0:1775765681.333359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765681/real 1775765681] req@ffff9d81901af100 x1862023066317696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765681.333363:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765681.333373:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765681.333383:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765681.333394:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066317696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765681.333403:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066318208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765681.333410:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765681.333422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765681.333431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066317696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765681.333441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765681.333463:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765681.333478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765681/real 1775765681] req@ffff9d81901ae680 x1862023066317824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765681.333490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066317824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765681.333499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765681.333503:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765681.333506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765681.333526:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765681.333541:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765681.333549:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765681/real 1775765681] req@ffff9d81901aed80 x1862023066317952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765681.333560:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066317952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765681.333568:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765681.333570:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765681.333573:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066317952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765681.333578:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765681.333592:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765681.333599:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765681/real 1775765681] req@ffff9d81901afb80 x1862023066318080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765681.333609:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066318080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765681.333617:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765681.333620:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765681.333623:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765681.333628:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765681.333641:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765681.333648:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765681/real 1775765681] req@ffff9d81901ad180 x1862023066318208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765681.333658:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066318208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765681.333665:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765681.333668:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765681.333670:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765681.973116:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765682.357030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765682.997118:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765683.381009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765684.021111:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765684.405120:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765685.045094:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765685.428927:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765686.069076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765686.453031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765686.453078:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765686.453087:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765686.453093:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765686.453103:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2517 00000100:00080000:0.0:1775765686.453115:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765686.453124:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765686.453166:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066318336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765686.453202:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765686.453206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765686.453214:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765686.453218:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2517 00000100:00080000:0.0:1775765686.453221:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765686.453225:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765686.453236:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765686.453251:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066318464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765686.453264:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066318336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765686.453268:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765686.453276:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765686.453278:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765686.453283:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2517 00000100:00080000:0.0:1775765686.453286:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765686.453290:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765686.453296:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765686.453310:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066318592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765686.453320:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765686.453324:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765686.453326:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765686.453331:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2517 00000100:00080000:0.0:1775765686.453335:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765686.453340:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765686.453345:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765686.453355:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765686.453377:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066318720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765686.453393:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765686.453398:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765686.453402:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765686.453405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765686/real 1775765686] req@ffff9d81901ad180 x1862023066318336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765686.453414:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2517 00000100:00080000:0.0:1775765686.453427:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765686.453435:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765686.453444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066318336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775765686.453446:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775765686.453469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775765686.453473:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066318848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765686.453484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765686.453497:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066318336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765686.453535:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765686.453567:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765686.453582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765686/real 1775765686] req@ffff9d81901afb80 x1862023066318464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765686.453596:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066318464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765686.453605:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765686.453609:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765686.453612:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765686.453619:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765686.453635:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765686.453644:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765686/real 1775765686] req@ffff9d81901aed80 x1862023066318592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765686.453656:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066318592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765686.453664:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765686.453667:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765686.453670:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765686.453679:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765686.453707:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765686.453725:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765686/real 1775765686] req@ffff9d81901ae680 x1862023066318720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765686.453756:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066318720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765686.453772:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765686.453778:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765686.453783:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765686.453792:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765686.453826:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765686.453844:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765686/real 1775765686] req@ffff9d81901af100 x1862023066318848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765686.453857:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066318848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765686.453867:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765686.453870:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765686.453873:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066318848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765687.093088:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765687.476948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765688.117029:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765688.500953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765689.141116:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765689.524955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765690.165191:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765690.548989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765691.189066:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765691.572939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765691.572957:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765691.572978:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765691.572982:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765691.572993:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2523 00000100:00080000:0.0:1775765691.573004:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765691.573011:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765691.573048:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066318976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765691.573076:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765691.573081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765691.573083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765691.573089:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2523 00000100:00080000:0.0:1775765691.573093:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765691.573097:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765691.573103:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066318976:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775765691.573107:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765691.573125:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066319104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765691.573138:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765691.573141:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765691.573143:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765691.573148:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2523 00000100:00080000:0.0:1775765691.573153:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765691.573156:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765691.573161:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765691.573167:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765691.573181:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066319232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765691.573191:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765691.573195:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765691.573197:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765691.573202:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765691/real 1775765691] req@ffff9d81901af100 x1862023066318976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765691.573203:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2523 00000100:00080000:0.0:1775765691.573210:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765691.573215:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765691.573225:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765691.573242:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066318976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765691.573245:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066319360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765691.573260:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765691.573266:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:0.0:1775765691.573267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765691.573271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066318976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765691.573274:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765691.573278:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765691.573284:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2523 00000100:00080000:0.0:1775765691.573288:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765691.573292:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765691.573297:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319104:192.168.202.105@tcp:38:kworker.0 00000008:00100000:0.0:1775765691.573299:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765691.573316:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066319488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765691.573330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765691.573346:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765691/real 1775765691] req@ffff9d81901ae680 x1862023066319104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765691.573359:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066319104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765691.573365:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765691.573367:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765691.573369:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765691.573373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765691.573387:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765691.573394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765691/real 1775765691] req@ffff9d81901aed80 x1862023066319232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765691.573402:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066319232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765691.573411:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765691.573413:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765691.573415:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765691.573418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765691.573428:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765691.573433:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765691/real 1775765691] req@ffff9d81901afb80 x1862023066319360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765691.573440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066319360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765691.573446:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765691.573448:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765691.573450:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765691.573459:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765691.573469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765691.573475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765691/real 1775765691] req@ffff9d81901ad180 x1862023066319488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765691.573482:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066319488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765691.573487:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765691.573489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765691.573492:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765692.213081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765692.596943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765693.237021:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765693.620984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765694.261050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765694.644967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765695.285027:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765695.668975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765696.309069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765696.693029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765696.693047:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765696.693054:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765696.693059:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765696.693069:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2528 00000100:00080000:1.0:1775765696.693080:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765696.693086:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765696.693125:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066319616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765696.693152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765696.693156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765696.693157:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765696.693161:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2528 00000100:00080000:1.0:1775765696.693164:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765696.693168:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765696.693178:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765696.693192:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066319744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765696.693193:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066319616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765696.693224:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765696.693227:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765696.693229:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765696.693233:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2528 00000100:00080000:1.0:1775765696.693235:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765696.693238:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765696.693243:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765696.693253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066319872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765696.693259:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765696.693262:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765696.693264:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765696.693266:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2528 00000100:00080000:1.0:1775765696.693269:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765696.693272:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765696.693277:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775765696.693280:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765696.693289:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066320000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765696.693296:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765696.693299:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765696.693300:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765696.693304:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2528 00000100:00080000:1.0:1775765696.693307:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765696.693309:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765696.693314:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765696.693325:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066320128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765696.693325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765696/real 1775765696] req@ffff9d8190551880 x1862023066319616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765696.693370:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066319616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765696.693380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765696.693383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765696.693387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066319616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765696.693405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765696.693424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765696.693432:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765696/real 1775765696] req@ffff9d8190550e00 x1862023066319744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765696.693442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066319744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765696.693448:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765696.693451:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765696.693453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765696.693457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765696.693469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765696.693475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765696/real 1775765696] req@ffff9d8190553100 x1862023066319872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765696.693484:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066319872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765696.693489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765696.693491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765696.693493:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066319872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765696.693497:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765696.693516:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765696.693527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765696/real 1775765696] req@ffff9d8190553480 x1862023066320000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765696.693539:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066320000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765696.693550:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765696.693554:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765696.693557:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765696.693564:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765696.693585:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765696.693597:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765696/real 1775765696] req@ffff9d8190551500 x1862023066320128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765696.693611:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066320128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765696.693620:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765696.693625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765696.693629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765697.333159:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765697.717019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765698.357046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765698.740976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765699.381015:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765699.764996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765700.405049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765700.788994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765701.429001:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765701.813015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765701.813015:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765701.813025:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765701.813030:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765701.813040:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2533 00000100:00080000:2.0:1775765701.813051:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765701.813058:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765701.813103:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066320256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765701.813123:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765701.813126:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765701.813127:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765701.813130:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2533 00000100:00080000:2.0:1775765701.813133:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765701.813135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765701.813142:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765701.813150:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066320384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765701.813158:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765701.813160:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765701.813162:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765701.813165:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2533 00000100:00080000:2.0:1775765701.813167:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765701.813169:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765701.813172:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765701.813182:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066320512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765701.813183:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066320256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765701.813201:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765701.813204:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765701.813207:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765701.813211:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2533 00000100:00080000:2.0:1775765701.813214:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765701.813218:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765701.813225:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765701.813238:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066320640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765701.813248:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765701.813250:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765701.813254:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765701.813256:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765701.813260:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2533 00000100:00080000:2.0:1775765701.813264:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765701.813267:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765701.813272:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765701.813282:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066320768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765701.813300:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765701/real 1775765701] req@ffff9d81984a0000 x1862023066320256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765701.813320:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066320256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765701.813331:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765701.813336:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765701.813340:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066320256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765701.813347:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765701.813366:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765701.813377:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765701/real 1775765701] req@ffff9d81984a2a00 x1862023066320384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765701.813390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066320384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765701.813399:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765701.813402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765701.813405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765701.813425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765701.813442:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765701.813452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765701/real 1775765701] req@ffff9d81984a2d80 x1862023066320512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765701.813464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066320512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765701.813474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765701.813477:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765701.813481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765701.813486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765701.813502:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765701.813510:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765701/real 1775765701] req@ffff9d81984a0700 x1862023066320640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765701.813523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066320640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765701.813539:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765701.813543:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765701.813546:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765701.813552:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765701.813566:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765701.813575:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765701/real 1775765701] req@ffff9d81984a3b80 x1862023066320768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765701.813586:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066320768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765701.813595:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765701.813599:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765701.813602:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066320768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765702.453095:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765702.836944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765703.477134:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765703.860991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765704.501100:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765704.884954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765705.525052:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765705.909196:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765706.549008:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765706.932948:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765706.932955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765706.932959:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765706.932965:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765706.932974:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2538 00000100:00080000:2.0:1775765706.932984:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765706.932992:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765706.933027:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066320896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765706.933058:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765706.933062:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765706.933064:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765706.933073:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2538 00000100:00080000:2.0:1775765706.933077:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765706.933081:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765706.933090:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765706.933105:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066321024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765706.933119:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765706.933119:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066320896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765706.933129:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765706.933132:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765706.933137:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2538 00000100:00080000:2.0:1775765706.933141:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765706.933145:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765706.933150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765706.933163:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066321152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765706.933174:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765706.933181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775765706.933181:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765706.933185:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765706.933191:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2538 00000100:00080000:2.0:1775765706.933197:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765706.933201:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765706.933207:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765706.933221:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066321280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765706.933226:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765706/real 1775765706] req@ffff9d81984a3480 x1862023066320896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765706.933246:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765706.933250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765706.933252:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765706.933253:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066320896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765706.933260:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2538 00000100:00080000:2.0:1775765706.933268:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765706.933269:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765706.933275:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775765706.933276:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:2.0:1775765706.933283:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765706.933283:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066320896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765706.933301:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066321408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765706.933304:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765706.933328:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765706.933338:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765706/real 1775765706] req@ffff9d81984a3b80 x1862023066321024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765706.933347:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066321024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765706.933352:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765706.933355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765706.933357:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765706.933360:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765706.933370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765706.933375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765706/real 1775765706] req@ffff9d81984a0700 x1862023066321152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765706.933382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066321152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765706.933387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765706.933388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765706.933390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765706.933393:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765706.933403:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765706.933409:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765706/real 1775765706] req@ffff9d81984a2d80 x1862023066321280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765706.933416:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066321280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765706.933422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765706.933424:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765706.933426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765706.933434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765706.933444:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765706.933449:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765706/real 1775765706] req@ffff9d81984a2a00 x1862023066321408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765706.933456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066321408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765706.933461:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765706.933463:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765706.933465:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765707.573065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765707.956988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765708.597084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765708.980990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765709.621026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765710.004967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765710.645092:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765711.028982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765711.669103:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765712.052984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765712.053038:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765712.053064:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765712.053067:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765712.053074:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2543 00000100:00080000:3.0:1775765712.053088:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765712.053098:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765712.053141:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066321536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765712.053176:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765712.053180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765712.053181:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765712.053186:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2543 00000100:00080000:3.0:1775765712.053189:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765712.053193:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765712.053203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765712.053216:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066321664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765712.053228:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765712.053231:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765712.053232:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765712.053247:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2543 00000100:00080000:3.0:1775765712.053251:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775765712.053252:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066321536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765712.053257:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765712.053263:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765712.053272:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066321792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765712.053280:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765712.053282:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765712.053283:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765712.053286:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2543 00000100:00080000:3.0:1775765712.053289:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765712.053292:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765712.053302:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765712.053310:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066321920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765712.053316:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765712.053319:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765712.053320:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765712.053322:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2543 00000100:00080000:3.0:1775765712.053325:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765712.053327:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765712.053331:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775765712.053333:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765712.053342:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066322048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765712.053376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765712/real 1775765712] req@ffff9d8188743800 x1862023066321536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765712.053390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066321536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765712.053398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765712.053401:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765712.053404:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066321536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765712.053409:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765712.053436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765712.053444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765712/real 1775765712] req@ffff9d8188742680 x1862023066321664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765712.053461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066321664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765712.053467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765712.053469:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765712.053471:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765712.053516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765712.053544:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765712.053553:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765712/real 1775765712] req@ffff9d8188742300 x1862023066321792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765712.053563:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066321792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765712.053570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765712.053572:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765712.053575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765712.053595:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765712.053607:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765712.053613:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765712/real 1775765712] req@ffff9d8188741c00 x1862023066321920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765712.053620:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066321920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765712.053626:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765712.053628:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765712.053629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066321920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765712.053633:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765712.053642:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765712.053647:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765712/real 1775765712] req@ffff9d8188742a00 x1862023066322048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765712.053654:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066322048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765712.053660:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765712.053662:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765712.053665:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765712.693046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765713.076987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765713.717090:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765714.100946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765714.741124:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765715.125004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765715.765048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765716.148936:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765716.789093:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765717.173014:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765717.173016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765717.173027:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765717.173033:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765717.173046:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2548 00000100:00080000:3.0:1775765717.173059:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765717.173069:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765717.173113:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066322176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765717.173151:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765717.173156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765717.173159:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765717.173165:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2548 00000100:00080000:3.0:1775765717.173170:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765717.173176:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765717.173188:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765717.173208:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066322176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765717.173226:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066322304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765717.173243:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765717.173248:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765717.173250:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765717.173256:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2548 00000100:00080000:3.0:1775765717.173261:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765717.173266:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765717.173273:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765717.173290:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066322432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765717.173304:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765717.173309:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765717.173312:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765717.173318:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2548 00000100:00080000:3.0:1775765717.173323:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775765717.173328:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765717.173334:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765717.173342:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765717.173372:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066322560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765717.173387:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765717.173392:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765717.173395:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775765717.173403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765717/real 1775765717] req@ffff9d8188740700 x1862023066322176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765717.173404:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2548 00000100:00080000:3.0:1775765717.173413:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765717.173423:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765717.173434:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765717.173444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066322176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765717.173453:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066322688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765717.173460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765717.173473:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765717.173483:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066322176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765717.173513:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765717.173545:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765717.173562:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765717/real 1775765717] req@ffff9d8188740380 x1862023066322304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765717.173581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066322304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765717.173593:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765717.173597:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765717.173602:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765717.173610:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765717.173631:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765717.173641:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765717/real 1775765717] req@ffff9d8188740000 x1862023066322432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765717.173651:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066322432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765717.173665:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765717.173667:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765717.173670:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765717.173675:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765717.173688:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765717.173695:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765717/real 1775765717] req@ffff9d8188742a00 x1862023066322560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765717.173704:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066322560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765717.173712:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765717.173714:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765717.173741:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765717.173746:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765717.173765:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765717.173774:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765717/real 1775765717] req@ffff9d8188741c00 x1862023066322688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765717.173784:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066322688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765717.173800:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765717.173802:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765717.173805:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765717.813027:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765718.197022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765718.837165:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765719.220985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765719.861088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765720.244961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765720.885077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765721.268961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765721.909093:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765722.292959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765722.292964:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765722.292995:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765722.292999:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765722.293009:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2553 00000100:00080000:2.0:1775765722.293018:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765722.293024:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765722.293060:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066322816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765722.293085:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765722.293088:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765722.293089:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765722.293092:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2553 00000100:00080000:2.0:1775765722.293094:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765722.293097:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765722.293105:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765722.293115:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066322944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765722.293123:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765722.293126:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765722.293127:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765722.293131:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2553 00000100:00080000:2.0:1775765722.293132:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765722.293135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765722.293138:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765722.293147:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066323072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765722.293148:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066322816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765722.293181:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765722.293184:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765722.293186:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765722.293190:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2553 00000100:00080000:2.0:1775765722.293195:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765722.293199:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765722.293206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765722.293219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066323200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765722.293232:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775765722.293233:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765722.293238:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765722.293241:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765722.293255:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2553 00000100:00080000:2.0:1775765722.293260:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765722.293268:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765722.293289:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765722.293296:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765722/real 1775765722] req@ffff9d81984a0000 x1862023066322816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765722.293301:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066323328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765722.293335:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066322816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765722.293347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765722.293351:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765722.293356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066322816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765722.293363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765722.293384:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765722.293396:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765722/real 1775765722] req@ffff9d81984a2680 x1862023066322944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765722.293409:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066322944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765722.293418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765722.293421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765722.293425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066322944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765722.293446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765722.293462:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765722.293471:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765722/real 1775765722] req@ffff9d81984a1f80 x1862023066323072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765722.293483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066323072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765722.293491:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765722.293495:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765722.293498:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765722.293504:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765722.293518:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765722.293526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765722/real 1775765722] req@ffff9d81984a1180 x1862023066323200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765722.293538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066323200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765722.293548:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765722.293551:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765722.293554:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765722.293560:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765722.293573:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765722.293581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765722/real 1775765722] req@ffff9d81984a1500 x1862023066323328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765722.293609:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066323328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765722.293620:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765722.293623:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765722.293627:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765722.933023:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765723.317052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765723.957001:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765724.340941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765724.981117:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765725.365009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765726.005095:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765726.388954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765727.029249:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765727.413008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765727.413055:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765727.413077:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765727.413082:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765727.413092:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2558 00000100:00080000:3.0:1775765727.413103:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765727.413110:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765727.413170:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066323456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765727.413211:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765727.413216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765727.413234:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765727.413239:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2558 00000100:00080000:3.0:1775765727.413243:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765727.413249:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765727.413259:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765727.413275:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066323584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765727.413290:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765727.413300:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066323456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765727.413300:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765727.413310:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765727.413322:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2558 00000100:00080000:3.0:1775765727.413327:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765727.413332:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765727.413339:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765727.413361:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066323712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765727.413374:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765727.413379:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765727.413381:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765727.413386:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2558 00000100:00080000:3.0:1775765727.413404:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765727.413409:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765727.413418:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765727.413431:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066323840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765727.413442:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765727.413447:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765727.413450:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765727.413459:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2558 00000100:00080000:3.0:1775765727.413467:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765727.413476:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765727.413484:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765727.413502:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765727.413510:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066323968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765727.413564:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765727.413599:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765727.413675:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765728.053055:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765728.437032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765729.077055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765729.461039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765730.101108:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765730.485221:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775765730.485442:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775765730.485513:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775765730.485535:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765730.485627:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765730.485641:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765730.485653:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765730.485667:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775765730.485714:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765727/real 1775765730] req@ffff9d8188741880 x1862023066323968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765730.485753:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066323968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765730.485772:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765730.485782:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765730.485789:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323968:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765730.485801:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765727/real 1775765730] req@ffff9d8188742680 x1862023066323584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765730.485824:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066323584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765730.485838:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765730.485844:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765730.485850:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765730.485861:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765727/real 1775765730] req@ffff9d8188743800 x1862023066323712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765730.485973:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066323712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765730.485989:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765730.485995:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765730.486001:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765730.486012:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765727/real 1775765730] req@ffff9d8188741f80 x1862023066323840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765730.486043:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066323840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765730.486059:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765730.486065:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765730.486071:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066323840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765730.486090:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765727/real 1775765730] req@ffff9d8188742300 x1862023066323456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765730.486114:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066323456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765730.486129:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765730.486138:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765730.486147:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066323456:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775765731.125061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765731.509026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765732.149086:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765732.532981:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765732.532990:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765732.532993:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765732.533002:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2564 00000100:00080000:3.0:1775765732.533027:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765732.533035:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765732.533050:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765732.533074:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066324096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765732.533103:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765732.533106:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765732.533107:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765732.533110:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2564 00000100:00080000:3.0:1775765732.533113:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765732.533116:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765732.533123:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765732.533134:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066324224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765732.533142:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765732.533144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765732.533146:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765732.533149:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2564 00000100:00080000:3.0:1775765732.533151:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765732.533153:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765732.533156:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765732.533165:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066324352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765732.533170:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066324096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765732.533176:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765732.533180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765732.533182:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765732.533185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2564 00000100:00080000:3.0:1775765732.533188:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765732.533191:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765732.533197:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765732.533210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066324480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765732.533221:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765732.533225:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765732.533227:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765732.533231:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2564 00000800:00000100:0.0:1775765732.533251:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765732.533254:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765732.533261:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765732.533270:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765732.533283:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066324608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765732.533305:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765732/real 1775765732] req@ffff9d8188740a80 x1862023066324096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765732.533326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066324096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765732.533337:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765732.533341:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765732.533346:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066324096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765732.533353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765732.533378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765732.533390:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765732/real 1775765732] req@ffff9d8188741500 x1862023066324224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765732.533400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066324224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765732.533406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765732.533408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765732.533411:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765732.533430:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765732.533461:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765732.533472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765732/real 1775765732] req@ffff9d8188741180 x1862023066324352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765732.533485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066324352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765732.533493:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765732.533497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765732.533500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765732.533506:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765732.533528:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765732.533539:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765732/real 1775765732] req@ffff9d8188743100 x1862023066324480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765732.533551:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066324480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765732.533566:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765732.533586:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765732.533590:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765732.533597:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765732.533619:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765732.533630:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765732/real 1775765732] req@ffff9d8188740e00 x1862023066324608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765732.533641:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066324608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765732.533650:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765732.533653:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765732.533656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765733.173172:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765733.557108:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765734.197145:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765734.581064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765735.221103:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765735.604980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765736.245065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765736.629037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765737.269135:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765737.652999:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765737.653016:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775765737.653023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765737.653024:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765737.653047:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2569 00000100:00080000:3.0:1775765737.653065:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765737.653078:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765737.653129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066324736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765737.653172:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765737.653177:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765737.653180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765737.653185:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2569 00000100:00080000:3.0:1775765737.653190:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765737.653196:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765737.653207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765737.653223:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066324864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765737.653237:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765737.653242:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765737.653245:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765737.653250:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2569 00000100:00080000:3.0:1775765737.653255:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765737.653259:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765737.653266:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765737.653272:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066324736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765737.653284:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066324992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765737.653307:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765737.653312:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765737.653315:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765737.653320:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2569 00000100:00080000:3.0:1775765737.653347:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765737.653351:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765737.653359:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775765737.653361:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765737.653384:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066325120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765737.653398:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765737.653402:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765737.653405:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765737.653412:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2569 00000100:00080000:3.0:1775765737.653419:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765737.653431:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765737.653432:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765737/real 1775765737] req@ffff9d8188743480 x1862023066324736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775765737.653448:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765737.653466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066324736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765737.653479:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066325248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765737.653553:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765737.653563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765737.653570:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066324736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765737.653582:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765737.653611:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765737.653627:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765737/real 1775765737] req@ffff9d8188742d80 x1862023066324864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765737.653643:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066324864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765737.653653:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765737.653657:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765737.653661:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765737.653688:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765737.653707:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765737.653718:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765737/real 1775765737] req@ffff9d8188743b80 x1862023066324992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765737.653732:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066324992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765737.653742:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765737.653745:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765737.653750:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066324992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765737.653757:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765737.653775:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765737.653785:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765737/real 1775765737] req@ffff9d8188740e00 x1862023066325120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765737.653798:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066325120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765737.653809:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765737.653812:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765737.653816:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765737.653822:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765737.653839:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765737.653849:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765737/real 1775765737] req@ffff9d8188743100 x1862023066325248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765737.653862:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066325248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765737.653872:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765737.653916:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765737.653920:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765738.293048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765738.677022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765739.317172:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765739.700993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765740.341107:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765740.725016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765741.365065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765741.748943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765742.389137:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765742.773055:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765742.773147:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765742.773156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765742.773162:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765742.773172:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2574 00000100:00080000:0.0:1775765742.773185:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765742.773193:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765742.773240:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066325376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765742.773278:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765742.773283:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765742.773285:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765742.773290:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2574 00000100:00080000:0.0:1775765742.773294:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765742.773297:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765742.773309:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765742.773324:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066325504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765742.773338:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765742.773343:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066325376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765742.773345:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765742.773350:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765742.773363:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2574 00000100:00080000:0.0:1775765742.773367:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765742.773371:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765742.773378:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765742.773396:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066325632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765742.773407:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765742.773411:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765742.773413:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765742.773417:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2574 00000100:00080000:0.0:1775765742.773422:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765742.773434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765742.773437:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765742.773447:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765742.773464:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066325760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765742.773504:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765742.773508:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765742.773511:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765742.773518:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2574 00000100:00080000:0.0:1775765742.773525:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765742.773525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765742/real 1775765742] req@ffff9d81901ad180 x1862023066325376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765742.773537:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765742.773551:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765742.773569:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066325376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765742.773575:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066325888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765742.773606:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765742.773623:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765742.773633:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066325376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765742.773671:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765742.773711:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765742.773729:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765742/real 1775765742] req@ffff9d81901afb80 x1862023066325504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765742.773745:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066325504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765742.773755:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765742.773759:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765742.773763:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765742.773769:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765742.773789:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765742.773799:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765742/real 1775765742] req@ffff9d81901aed80 x1862023066325632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765742.773812:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066325632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765742.773829:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765742.773833:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765742.773836:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765742.773842:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765742.773859:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765742.773885:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765742/real 1775765742] req@ffff9d81901ae680 x1862023066325760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765742.773937:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066325760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765742.773949:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765742.773953:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765742.773957:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765742.773965:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765742.773991:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765742.774003:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765742/real 1775765742] req@ffff9d81901af100 x1862023066325888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765742.774018:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066325888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765742.774028:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765742.774032:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765742.774036:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066325888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765743.413090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765743.797012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765744.437209:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765744.821010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765745.461116:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765745.844981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765746.485058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765746.869025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765747.509073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765747.893056:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765747.893089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765747.893102:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765747.893109:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765747.893125:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2579 00000100:00080000:3.0:1775765747.893141:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765747.893154:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765747.893209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066326016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765747.893261:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765747.893268:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765747.893271:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765747.893279:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2579 00000100:00080000:3.0:1775765747.893288:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765747.893296:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765747.893313:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765747.893321:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066326016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765747.893344:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066326144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765747.893367:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765747.893374:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765747.893377:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765747.893385:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2579 00000100:00080000:3.0:1775765747.893394:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775765747.893405:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765747.893407:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765747.893421:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765747.893485:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066326272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765747.893500:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765747.893507:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775765747.893509:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765747/real 1775765747] req@ffff9d8188741180 x1862023066326016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765747.893511:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765747.893526:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2579 00000100:00080000:3.0:1775765747.893537:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765747.893544:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765747.893552:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765747.893556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066326016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765747.893571:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066326400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765747.893586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765747.893586:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765747.893598:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765747.893601:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765747.893604:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775765747.893611:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2579 00000100:00100000:0.0:1775765747.893618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066326016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765747.893620:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765747.893645:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765747.893651:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765747.893659:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765747.893668:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066326528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775765747.893698:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765747.893719:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765747/real 1775765747] req@ffff9d8188741500 x1862023066326144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765747.893737:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066326144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765747.893748:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765747.893752:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765747.893757:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765747.893763:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765747.893782:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765747.893793:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765747/real 1775765747] req@ffff9d8188740a80 x1862023066326272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765747.893807:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066326272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765747.893817:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765747.893820:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765747.893824:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765747.893831:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765747.893849:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765747.893859:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765747/real 1775765747] req@ffff9d8188742300 x1862023066326400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765747.893905:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066326400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765747.893918:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765747.893922:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765747.893927:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765747.893945:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765747.893968:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765747.893981:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765747/real 1775765747] req@ffff9d8188741f80 x1862023066326528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765747.893995:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066326528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765747.894013:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765747.894017:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765747.894021:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765748.533087:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765748.917051:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765749.557106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765749.941046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765750.581026:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765750.965000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765751.605094:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765751.989025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765752.629171:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765753.013001:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765753.013015:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765753.013021:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765753.013051:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2584 00000100:00080000:2.0:1775765753.013066:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775765753.013072:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765753.013076:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765753.013148:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066326656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765753.013200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765753.013206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765753.013208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765753.013211:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2584 00000100:00080000:2.0:1775765753.013214:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765753.013217:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765753.013228:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765753.013239:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066326784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765753.013249:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765753.013253:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765753.013255:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765753.013259:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2584 00000100:00080000:2.0:1775765753.013261:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765753.013263:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765753.013267:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765753.013274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066326912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765753.013282:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765753.013286:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765753.013287:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765753.013293:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2584 00000100:00080000:2.0:1775765753.013299:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775765753.013303:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066326656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765753.013304:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765753.013311:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765753.013349:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066327040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765753.013357:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765753.013360:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765753.013361:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765753.013364:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2584 00000100:00080000:2.0:1775765753.013368:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765753.013371:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765753.013389:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765753.013400:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066327168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775765753.013462:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765753.013538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765753/real 1775765753] req@ffff9d81984a2300 x1862023066326656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765753.013571:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066326656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765753.013586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765753.013592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765753.013597:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066326656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765753.013608:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765753.013642:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765753.013669:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765753/real 1775765753] req@ffff9d81984a1880 x1862023066326784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765753.013684:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066326784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765753.013704:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765753.013708:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765753.013712:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765753.013753:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765753.013774:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765753.013785:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765753/real 1775765753] req@ffff9d81984a3800 x1862023066326912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765753.013809:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066326912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765753.013819:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765753.013822:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765753.013826:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066326912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765753.013833:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765753.013852:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765753.013862:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765753/real 1775765753] req@ffff9d81984a3100 x1862023066327040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765753.013924:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066327040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765753.013937:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765753.013941:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765753.013946:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765753.013954:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765753.013988:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765753.014001:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765753/real 1775765753] req@ffff9d81984a0e00 x1862023066327168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765753.014015:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066327168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765753.014035:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765753.014039:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765753.014043:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765753.653045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765754.037023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765754.677121:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765755.061032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765755.701016:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765756.084980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765756.725007:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765757.109003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765757.749186:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765758.132990:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765758.133005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765758.133198:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765758.133204:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765758.133227:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2589 00000100:00080000:2.0:1775765758.133242:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765758.133280:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765758.134649:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066327296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765758.134692:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765758.134696:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765758.134698:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765758.134702:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2589 00000100:00080000:2.0:1775765758.134707:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765758.134713:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765758.134726:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765758.134749:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066327424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765758.134750:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066327296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765758.134775:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765758.134778:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765758.134779:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765758.134786:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2589 00000100:00080000:2.0:1775765758.134789:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765758.134793:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765758.134797:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765758.134810:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066327552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765758.134818:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765758.134821:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765758.134822:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765758.134825:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2589 00000100:00080000:2.0:1775765758.134828:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765758.134832:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765758.134836:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765758.134843:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066327680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775765758.134860:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765758.134860:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765758.134870:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765758.134873:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765758.134905:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2589 00000100:00080000:2.0:1775765758.134910:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765758.134914:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765758.134921:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765758.134949:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066327808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765758.134994:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765758/real 1775765758] req@ffff9d81984a0a80 x1862023066327296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765758.135023:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066327296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765758.135046:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765758.135051:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765758.135055:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066327296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765758.135104:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765758.135154:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765758.135164:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765758/real 1775765758] req@ffff9d81984a0e00 x1862023066327424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765758.135173:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066327424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765758.135180:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765758.135182:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765758.135185:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765758.135189:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765758.135224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765758.135238:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765758/real 1775765758] req@ffff9d81984a3100 x1862023066327552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765758.135255:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066327552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765758.135268:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765758.135273:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765758.135278:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765758.135288:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765758.135343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765758.135375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765758/real 1775765758] req@ffff9d81984a3800 x1862023066327680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765758.135385:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066327680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765758.135392:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765758.135394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765758.135396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765758.135401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765758.135416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765758.135422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765758/real 1775765758] req@ffff9d81984a1880 x1862023066327808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765758.135430:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066327808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765758.135436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765758.135438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765758.135440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066327808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765758.773110:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765759.157042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765759.797038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765760.181000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765760.821074:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765761.204992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765761.845159:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765762.229008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765762.869018:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765763.253095:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765763.253120:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765763.253124:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765763.253134:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2594 00000100:00080000:0.0:1775765763.253144:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765763.253151:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765763.253193:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:0.0:1775765763.253202:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066327936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765763.253233:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765763.253237:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765763.253238:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765763.253257:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2594 00000100:00080000:0.0:1775765763.253260:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765763.253262:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765763.253270:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765763.253296:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066328064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765763.253306:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765763.253310:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765763.253312:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765763.253316:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2594 00000100:00080000:0.0:1775765763.253321:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765763.253325:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765763.253331:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765763.253343:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066327936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765763.253345:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066328192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765763.253359:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765763.253362:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765763.253363:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765763.253366:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2594 00000100:00080000:0.0:1775765763.253369:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765763.253371:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765763.253375:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765763.253383:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066328320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765763.253389:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765763.253391:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765763.253392:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765763.253394:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2594 00000100:00080000:0.0:1775765763.253396:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765763.253398:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765763.253401:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765763.253408:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066328448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765763.253435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765763.253468:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765763/real 1775765763] req@ffff9d81901af100 x1862023066327936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765763.253482:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066327936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765763.253489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765763.253492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765763.253495:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066327936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765763.253499:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765763.253512:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765763.253536:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765763/real 1775765763] req@ffff9d81901ae680 x1862023066328064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765763.253546:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066328064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765763.253552:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765763.253554:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765763.253557:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765763.253592:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765763.253623:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765763.253637:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765763/real 1775765763] req@ffff9d81901aed80 x1862023066328192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765763.253645:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066328192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765763.253650:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765763.253652:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765763.253654:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765763.253658:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765763.253668:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765763.253673:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765763/real 1775765763] req@ffff9d81901afb80 x1862023066328320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765763.253680:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066328320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765763.253686:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765763.253687:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765763.253689:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765763.253692:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765763.253702:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765763.253707:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765763/real 1775765763] req@ffff9d81901ad180 x1862023066328448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765763.253714:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066328448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765763.253719:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765763.253721:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765763.253723:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765763.893026:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765764.276982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765764.917133:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765765.300960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765765.941062:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765766.324961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765766.965030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765767.349001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765767.989040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765768.372966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765768.373035:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765768.373042:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765768.373047:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765768.373059:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2599 00000100:00080000:0.0:1775765768.373069:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765768.373077:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765768.373119:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066328576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765768.373147:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765768.373150:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765768.373152:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765768.373162:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2599 00000100:00080000:0.0:1775765768.373165:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765768.373168:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765768.373178:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765768.373192:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066328704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765768.373205:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066328576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765768.373206:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765768.373211:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765768.373212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765768.373221:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2599 00000100:00080000:0.0:1775765768.373224:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765768.373265:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765768.373269:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765768.373281:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066328832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765768.373299:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765768.373303:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765768.373305:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765768.373309:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2599 00000100:00080000:0.0:1775765768.373318:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765768.373322:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765768.373329:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765768.373330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765768.373347:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066328960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765768.373358:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765768.373360:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765768.373362:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765768.373366:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2599 00000100:00080000:0.0:1775765768.373370:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765768.373373:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765768.373377:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765768/real 1775765768] req@ffff9d81901ad180 x1862023066328576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775765768.373380:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765768.373396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066328576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765768.373403:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066329088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765768.373407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765768.373416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765768.373421:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066328576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765768.373441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765768.373461:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765768.373469:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765768/real 1775765768] req@ffff9d81901afb80 x1862023066328704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765768.373478:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066328704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765768.373485:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765768.373487:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765768.373490:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765768.373493:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765768.373504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765768.373510:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765768/real 1775765768] req@ffff9d81901aed80 x1862023066328832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765768.373518:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066328832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765768.373524:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765768.373525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765768.373527:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765768.373531:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765768.373542:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765768.373548:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765768/real 1775765768] req@ffff9d81901ae680 x1862023066328960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765768.373556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066328960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765768.373562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765768.373564:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765768.373567:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066328960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765768.373571:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765768.373581:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765768.373587:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765768/real 1775765768] req@ffff9d81901af100 x1862023066329088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765768.373594:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066329088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765768.373600:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765768.373602:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765768.373604:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765769.013114:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765769.397008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765770.037122:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765770.420996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765771.061080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765771.445061:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765772.085038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765772.468998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765773.109076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765773.493007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765773.493086:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765773.493097:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765773.493103:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765773.493117:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2605 00000100:00080000:3.0:1775765773.493133:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765773.493143:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765773.493219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066329216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765773.493259:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765773.493263:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765773.493265:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765773.493270:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2605 00000100:00080000:3.0:1775765773.493274:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765773.493279:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765773.493292:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765773.493306:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066329344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765773.493315:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066329216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765773.493320:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765773.493332:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765773.493333:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765773.493339:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2605 00000100:00080000:3.0:1775765773.493342:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765773.493345:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765773.493350:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765773.493361:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066329472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765773.493369:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765773.493390:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765773.493392:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765773.493396:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2605 00000100:00080000:3.0:1775765773.493415:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765773.493420:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775765773.493427:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765773.493427:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765773.493446:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066329600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765773.493461:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765773.493466:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765773.493469:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765773.493475:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2605 00000100:00080000:3.0:1775765773.493482:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775765773.493489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765773/real 1775765773] req@ffff9d8188743800 x1862023066329216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765773.493489:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765773.493498:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765773.493535:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066329728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765773.493541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066329216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765773.493566:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765773.493572:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765773.493576:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066329216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765773.493603:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765773.493629:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765773.493646:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765773/real 1775765773] req@ffff9d8188742680 x1862023066329344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765773.493664:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066329344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765773.493677:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765773.493682:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765773.493702:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765773.493710:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765773.493733:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765773.493744:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765773/real 1775765773] req@ffff9d8188741880 x1862023066329472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765773.493768:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066329472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765773.493779:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765773.493783:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765773.493787:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765773.493795:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765773.493820:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765773.493833:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765773/real 1775765773] req@ffff9d8188741c00 x1862023066329600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765773.493849:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066329600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765773.493861:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765773.493865:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765773.493869:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765773.493878:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765773.493901:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765773.493911:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765773/real 1775765773] req@ffff9d8188742a00 x1862023066329728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765773.493959:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066329728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765773.493974:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765773.493977:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765773.493980:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765774.133216:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765774.517015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765775.157097:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765775.541039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765776.181111:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765776.564985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765777.205181:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765777.589034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765778.229206:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765778.612983:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765778.612992:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775765778.612996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765778.612996:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765778.613008:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2610 00000100:00080000:3.0:1775765778.613018:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765778.613024:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765778.613059:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066329856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765778.613087:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765778.613090:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765778.613092:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765778.613095:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2610 00000100:00080000:3.0:1775765778.613098:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765778.613102:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765778.613110:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765778.613118:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066329984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765778.613125:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765778.613128:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765778.613148:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765778.613151:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2610 00000100:00080000:3.0:1775765778.613154:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765778.613157:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765778.613161:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765778.613170:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066330112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765778.613177:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765778.613180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765778.613182:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765778.613185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2610 00000100:00080000:3.0:1775765778.613188:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765778.613190:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765778.613194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765778.613206:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066330240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765778.613207:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066329856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765778.613230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765778.613232:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765778.613234:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765778.613237:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2610 00000100:00080000:3.0:1775765778.613239:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765778.613242:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765778.613245:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765778.613255:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066330368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775765778.613302:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765778.613369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765778/real 1775765778] req@ffff9d8188740000 x1862023066329856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765778.613390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066329856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765778.613402:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765778.613408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765778.613412:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066329856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765778.613421:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765778.613449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765778.613463:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765778/real 1775765778] req@ffff9d8188740380 x1862023066329984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765778.613477:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066329984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765778.613492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765778.613496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765778.613499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066329984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765778.613506:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765778.613528:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765778.613541:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765778/real 1775765778] req@ffff9d8188740700 x1862023066330112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765778.613554:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066330112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765778.613564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765778.613567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765778.613571:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765778.613600:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765778.613661:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765778.613677:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765778/real 1775765778] req@ffff9d8188742a00 x1862023066330240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765778.613692:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066330240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765778.613703:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765778.613707:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765778.613712:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765778.613719:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765778.613742:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765778.613755:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765778/real 1775765778] req@ffff9d8188741c00 x1862023066330368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765778.613770:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066330368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765778.613793:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765778.613797:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765778.613801:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765779.253248:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765779.637069:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765780.277112:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765780.660949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765781.301126:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765781.684985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765782.325159:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765782.708997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765783.349043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765783.732978:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765783.732979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765783.732987:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765783.732991:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765783.732998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2615 00000100:00080000:3.0:1775765783.733008:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765783.733013:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765783.733046:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066330496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765783.733069:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765783.733072:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765783.733073:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765783.733076:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2615 00000100:00080000:3.0:1775765783.733078:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765783.733095:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765783.733103:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765783.733113:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066330624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765783.733121:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765783.733124:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765783.733127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775765783.733129:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066330496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765783.733133:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2615 00000100:00080000:3.0:1775765783.733139:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765783.733141:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765783.733144:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765783.733152:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066330752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765783.733158:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765783.733160:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765783.733161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765783.733164:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2615 00000100:00080000:3.0:1775765783.733184:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765783.733189:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775765783.733192:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765783.733199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765783.733214:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066330880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765783.733224:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765783.733228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765783.733230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765783.733234:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2615 00000100:00100000:0.0:1775765783.733235:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765783/real 1775765783] req@ffff9d8188741880 x1862023066330496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765783.733238:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765783.733246:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765783.733252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765783.733258:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066330496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765783.733263:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066331008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765783.733274:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765783.733280:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765783.733283:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066330496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765783.733302:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765783.733317:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765783.733325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765783/real 1775765783] req@ffff9d8188742680 x1862023066330624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765783.733333:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066330624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765783.733338:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765783.733341:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765783.733343:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765783.733346:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765783.733356:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765783.733361:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765783/real 1775765783] req@ffff9d8188743800 x1862023066330752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765783.733368:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066330752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765783.733373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765783.733375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765783.733377:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765783.733380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765783.733390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765783.733395:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765783/real 1775765783] req@ffff9d8188741f80 x1862023066330880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765783.733403:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066330880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765783.733409:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765783.733410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765783.733412:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066330880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765783.733416:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765783.733425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765783.733430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765783/real 1775765783] req@ffff9d8188742300 x1862023066331008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765783.733437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066331008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765783.733443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765783.733444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765783.733446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765784.373094:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765784.756971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765785.397091:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765785.781024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765786.421113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765786.805013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765787.445125:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765787.829020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765788.469250:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765788.853122:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765788.853131:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765788.853139:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765788.853143:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765788.853152:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2620 00000100:00080000:0.0:1775765788.853163:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765788.853170:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765788.853216:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066331136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765788.853245:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765788.853248:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765788.853250:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765788.853253:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2620 00000100:00080000:0.0:1775765788.853256:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765788.853259:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765788.853268:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765788.853278:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066331264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765788.853287:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765788.853290:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765788.853292:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765788.853296:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2620 00000100:00080000:0.0:1775765788.853298:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765788.853302:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765788.853306:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765788.853315:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066331392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765788.853324:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765788.853332:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765788.853335:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765788.853336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066331136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765788.853341:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2620 00000100:00080000:0.0:1775765788.853350:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765788.853357:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765788.853362:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765788.853370:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066331520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765788.853379:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765788.853383:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765788.853384:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765788.853388:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2620 00000100:00080000:0.0:1775765788.853390:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765788.853393:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765788.853397:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765788.853416:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066331648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765788.853511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765788.853605:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765788/real 1775765788] req@ffff9d81901af100 x1862023066331136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765788.853651:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066331136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765788.853667:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765788.853673:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765788.853680:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066331136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765788.853693:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765788.853745:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765788.853768:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765788/real 1775765788] req@ffff9d81901ae680 x1862023066331264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765788.853823:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066331264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765788.853834:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765788.853839:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765788.853843:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765788.853890:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765788.853914:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765788.853989:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765788/real 1775765788] req@ffff9d81901aed80 x1862023066331392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765788.854011:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066331392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765788.854026:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765788.854069:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765788.854074:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765788.854092:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765788.854138:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765788.854152:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765788/real 1775765788] req@ffff9d81901afb80 x1862023066331520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765788.854165:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066331520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765788.854185:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765788.854189:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765788.854193:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765788.854200:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765788.854224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765788.854236:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765788/real 1775765788] req@ffff9d81901ad180 x1862023066331648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765788.854250:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066331648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765788.854260:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765788.854264:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765788.854267:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765789.493144:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765789.877065:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765790.517152:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765790.901003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765791.541210:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765791.924956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765792.565048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765792.949012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765793.589122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765793.972974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765793.973028:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765793.973033:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765793.973037:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765793.973044:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2625 00000100:00080000:3.0:1775765793.973053:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765793.973058:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765793.973092:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066331776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765793.973122:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765793.973125:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765793.973126:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765793.973129:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2625 00000100:00080000:3.0:1775765793.973131:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765793.973135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765793.973149:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765793.973153:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066331776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765793.973161:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066331904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765793.973169:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765793.973172:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765793.973173:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765793.973176:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2625 00000100:00080000:3.0:1775765793.973178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765793.973180:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765793.973184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765793.973192:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066332032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765793.973199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765793.973201:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765793.973202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765793.973204:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2625 00000100:00080000:3.0:1775765793.973208:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765793.973210:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765793.973214:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765793.973221:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066332160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765793.973227:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765793.973230:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765793.973231:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765793.973235:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2625 00000100:00080000:3.0:1775765793.973237:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765793.973240:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765793.973244:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765793.973252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066332288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765793.973265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765793.973292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765793.973308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765793.973322:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765794.613154:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765794.997107:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765795.637018:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765796.021097:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765796.661096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765797.045129:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775765797.045233:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775765797.045282:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775765797.045301:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765797.045346:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765797.045355:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765797.045363:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775765797.045371:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775765797.045422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765793/real 1775765797] req@ffff9d8188741500 x1862023066331904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765797.045448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066331904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765797.045460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765797.045465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765797.045470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066331904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765797.045489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765793/real 1775765797] req@ffff9d8188741180 x1862023066332032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765797.045503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066332032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765797.045510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765797.045514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765797.045517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765797.045524:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765793/real 1775765797] req@ffff9d8188743100 x1862023066332160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765797.045534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066332160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765797.045543:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765797.045546:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765797.045549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765797.045555:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765793/real 1775765797] req@ffff9d8188740e00 x1862023066332288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765797.045574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066332288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765797.045581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765797.045584:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765797.045587:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332288:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765797.045593:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765793/real 1775765797] req@ffff9d8188740a80 x1862023066331776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765797.045605:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066331776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765797.045613:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765797.045617:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765797.045620:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066331776:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775765797.685128:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765798.069093:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765798.709233:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765799.093049:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765799.093061:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765799.093068:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765799.093080:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2630 00000100:00080000:3.0:1775765799.093095:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765799.093104:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765799.093133:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775765799.093151:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066332416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765799.093193:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765799.093198:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765799.093201:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765799.093206:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2630 00000100:00080000:3.0:1775765799.093210:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765799.093214:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765799.093226:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765799.093243:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066332544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765799.093274:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765799.093279:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066332416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765799.093280:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765799.093284:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765799.093298:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2630 00000100:00080000:3.0:1775765799.093302:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765799.093306:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765799.093312:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765799.093332:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066332672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765799.093354:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765799.093358:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765799.093361:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765799.093364:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2630 00000100:00080000:3.0:1775765799.093393:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765799.093398:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765799.093406:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775765799.093414:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765799.093439:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066332800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765799.093457:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765799.093464:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765799.093468:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765799.093477:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2630 00000100:00080000:3.0:1775765799.093486:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765799.093494:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765799.093504:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765799.093514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765799/real 1775765799] req@ffff9d8188743b80 x1862023066332416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765799.093536:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066332928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765799.093567:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066332416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765799.093588:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765799.093598:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765799.093607:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066332416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765799.093648:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765799.093698:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765799.093728:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765799/real 1775765799] req@ffff9d8188742d80 x1862023066332544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765799.093754:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066332544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765799.093772:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765799.093779:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765799.093786:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765799.093799:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765799.093842:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765799.093865:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765799/real 1775765799] req@ffff9d8188743480 x1862023066332672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765799.093992:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066332672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765799.094025:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765799.094032:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765799.094039:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765799.094051:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765799.094109:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765799.094133:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765799/real 1775765799] req@ffff9d8188740a80 x1862023066332800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765799.094156:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066332800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765799.094174:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765799.094180:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765799.094187:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765799.094199:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765799.094241:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765799.094263:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765799/real 1775765799] req@ffff9d8188740e00 x1862023066332928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765799.094288:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066332928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765799.094305:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765799.094311:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765799.094318:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066332928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765799.733123:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765800.117191:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765800.757164:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765801.141054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765801.781211:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765802.165066:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765802.805084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765803.189182:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765803.829094:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765804.213046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765804.213048:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765804.213065:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765804.213069:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765804.213083:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2635 00000100:00080000:0.0:1775765804.213092:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765804.213101:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765804.213142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066333056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765804.213167:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765804.213170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765804.213171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765804.213174:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2635 00000100:00080000:0.0:1775765804.213176:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765804.213178:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765804.213186:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765804.213195:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066333184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765804.213203:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765804.213204:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066333056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765804.213214:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765804.213215:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765804.213219:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2635 00000100:00080000:0.0:1775765804.213222:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765804.213224:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765804.213228:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765804.213238:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066333312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765804.213244:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765804.213246:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765804.213247:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765804.213250:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2635 00000100:00080000:0.0:1775765804.213252:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765804.213254:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765804.213258:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765804.213262:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765804.213269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066333440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765804.213276:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765804.213279:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765804.213280:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765804.213282:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2635 00000100:00080000:0.0:1775765804.213285:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765804.213287:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765804.213290:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765804.213298:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066333568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765804.213308:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765804/real 1775765804] req@ffff9d81901ad180 x1862023066333056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765804.213326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066333056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765804.213334:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765804.213337:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765804.213340:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066333056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765804.213380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765804.213409:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765804.213424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765804/real 1775765804] req@ffff9d81901afb80 x1862023066333184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765804.213440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066333184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765804.213450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765804.213454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765804.213458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765804.213465:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765804.213501:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765804.213511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765804/real 1775765804] req@ffff9d81901aed80 x1862023066333312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765804.213527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066333312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765804.213552:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765804.213558:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765804.213563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765804.213572:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765804.213628:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765804.213665:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765804/real 1775765804] req@ffff9d81901ae680 x1862023066333440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765804.213681:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066333440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765804.213690:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765804.213693:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765804.213696:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765804.213701:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765804.213720:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765804.213729:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765804/real 1775765804] req@ffff9d81901af100 x1862023066333568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765804.213738:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066333568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765804.213746:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765804.213749:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765804.213752:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765804.853118:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765805.237009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765805.877079:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765806.261072:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765806.901129:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765807.284989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765807.925237:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765808.309052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765808.949048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765809.332988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765809.332991:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765809.333002:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765809.333006:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765809.333015:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2640 00000100:00080000:3.0:1775765809.333025:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765809.333030:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765809.333062:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066333696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765809.333089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765809.333091:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765809.333093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765809.333095:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2640 00000100:00080000:3.0:1775765809.333098:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765809.333100:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765809.333108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765809.333117:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066333824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765809.333126:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765809.333129:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066333696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765809.333130:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765809.333136:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765809.333151:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2640 00000100:00080000:3.0:1775765809.333154:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765809.333157:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765809.333160:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765809.333174:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066333952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765809.333184:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765809.333188:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765809.333189:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765809.333196:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2640 00000100:00080000:3.0:1775765809.333212:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765809.333217:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775765809.333222:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775765809.333224:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765809.333237:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066334080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765809.333247:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765809.333250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765809.333251:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765809.333254:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2640 00000100:00080000:3.0:1775765809.333257:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765809.333260:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765809.333264:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765809.333271:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066334208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765809.333289:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765809/real 1775765809] req@ffff9d8188743100 x1862023066333696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765809.333314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066333696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765809.333323:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765809.333326:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765809.333330:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066333696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765809.333354:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765809.333372:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765809.333381:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765809/real 1775765809] req@ffff9d8188741180 x1862023066333824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765809.333389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066333824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765809.333395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765809.333398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765809.333400:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765809.333406:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765809.333417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765809.333423:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765809/real 1775765809] req@ffff9d8188741500 x1862023066333952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765809.333431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066333952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765809.333436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765809.333438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765809.333440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066333952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765809.333444:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765809.333455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765809.333461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765809/real 1775765809] req@ffff9d8188742300 x1862023066334080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765809.333468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066334080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765809.333475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765809.333476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765809.333478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765809.333482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765809.333491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765809.333496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765809/real 1775765809] req@ffff9d8188741f80 x1862023066334208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765809.333504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066334208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765809.333510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765809.333512:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765809.333513:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765809.973127:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765810.356936:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765810.997048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765811.380984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765812.021063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765812.405013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765813.045117:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765813.428992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765814.069125:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765814.453014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765814.453082:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765814.453092:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765814.453098:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765814.453110:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2645 00000100:00080000:0.0:1775765814.453143:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765814.453153:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765814.453233:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066334336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765814.453278:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765814.453283:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765814.453285:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765814.453290:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2645 00000100:00080000:0.0:1775765814.453294:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765814.453300:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765814.453315:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765814.453335:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066334464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765814.453337:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066334336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765814.453361:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765814.453365:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765814.453367:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765814.453373:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2645 00000100:00080000:0.0:1775765814.453377:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765814.453381:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765814.453387:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765814.453402:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066334592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765814.453412:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765814.453416:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765814.453418:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765814.453423:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2645 00000100:00080000:0.0:1775765814.453428:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765814.453432:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765814.453438:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765814.453443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765814.453478:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066334720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765814.453490:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765814.453496:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765814.453500:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765814.453506:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2645 00000100:00080000:0.0:1775765814.453514:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765814.453519:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765814.453541:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765814.453556:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066334848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765814.453576:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765814/real 1775765814] req@ffff9d81901af100 x1862023066334336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765814.453605:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066334336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765814.453619:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765814.453624:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765814.453629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066334336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765814.453659:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765814.453686:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765814.453701:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765814/real 1775765814] req@ffff9d81901ae680 x1862023066334464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765814.453715:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066334464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765814.453724:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765814.453728:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765814.453732:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765814.453738:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765814.453774:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765814.453784:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765814/real 1775765814] req@ffff9d81901aed80 x1862023066334592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765814.453821:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066334592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765814.453830:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765814.453833:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765814.453837:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765814.453844:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765814.453864:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765814.453901:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765814/real 1775765814] req@ffff9d81901afb80 x1862023066334720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765814.453916:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066334720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765814.453927:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765814.453931:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765814.453934:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765814.453941:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765814.453961:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765814.453971:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765814/real 1775765814] req@ffff9d81901ad180 x1862023066334848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765814.453991:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066334848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765814.454000:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765814.454003:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765814.454006:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066334848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765815.093052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765815.476992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765816.117106:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765816.501014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765817.141073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765817.524965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765818.165095:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765818.549027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765819.189014:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765819.572948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765819.572966:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765819.572972:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765819.572976:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765819.572984:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2651 00000100:00080000:0.0:1775765819.572993:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765819.572999:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765819.573044:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066334976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765819.573073:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765819.573077:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765819.573080:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765819.573084:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2651 00000100:00080000:0.0:1775765819.573088:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765819.573104:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765819.573115:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765819.573132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066334976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765819.573134:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066335104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765819.573150:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765819.573153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765819.573155:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765819.573159:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2651 00000100:00080000:0.0:1775765819.573163:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765819.573166:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765819.573171:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765819.573181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066335232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765819.573188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765819.573192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765819.573194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775765819.573194:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765819.573201:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2651 00000100:00080000:0.0:1775765819.573205:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765819.573208:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765819.573213:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765819.573222:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066335360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765819.573255:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765819.573257:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765819.573259:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765819.573264:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2651 00000100:00100000:3.0:1775765819.573268:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765819/real 1775765819] req@ffff9d81901ad180 x1862023066334976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765819.573269:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765819.573277:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765819.573285:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765819.573297:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066335488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765819.573300:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066334976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765819.573324:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765819.573328:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765819.573332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066334976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765819.573353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765819.573373:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765819.573398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765819/real 1775765819] req@ffff9d81901afb80 x1862023066335104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765819.573407:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066335104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765819.573414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765819.573416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765819.573419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765819.573423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765819.573435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765819.573441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765819/real 1775765819] req@ffff9d81901aed80 x1862023066335232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765819.573449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066335232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765819.573456:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765819.573458:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765819.573460:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765819.573463:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765819.573475:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765819.573481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765819/real 1775765819] req@ffff9d81901ae680 x1862023066335360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765819.573489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066335360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765819.573496:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765819.573498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765819.573500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765819.573504:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765819.573514:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765819.573520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765819/real 1775765819] req@ffff9d81901af100 x1862023066335488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765819.573527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066335488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765819.573533:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765819.573535:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765819.573538:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765820.213087:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765820.596956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765821.237074:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765821.620940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765822.261106:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765822.645089:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765823.285137:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765823.668975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765824.309168:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765824.692959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765824.692970:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765824.692976:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765824.692980:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765824.692989:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2656 00000100:00080000:0.0:1775765824.692999:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765824.693021:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765824.693058:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066335616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765824.693108:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765824.693112:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765824.693114:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765824.693118:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2656 00000100:00080000:0.0:1775765824.693122:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765824.693127:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765824.693137:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765824.693150:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066335616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765824.693155:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066335744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765824.693170:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765824.693173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765824.693191:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765824.693195:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2656 00000100:00080000:0.0:1775765824.693197:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765824.693201:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765824.693206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765824.693216:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066335872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765824.693224:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765824.693227:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765824.693228:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775765824.693234:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765824.693235:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2656 00000100:00080000:0.0:1775765824.693240:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765824.693244:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765824.693248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765824.693257:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066336000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765824.693264:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765824.693267:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765824.693268:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765824.693272:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2656 00000100:00080000:0.0:1775765824.693276:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765824.693280:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765824.693283:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765824.693294:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066336128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765824.693303:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765824/real 1775765824] req@ffff9d81901af100 x1862023066335616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765824.693322:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066335616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765824.693331:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765824.693336:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765824.693342:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066335616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765824.693373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765824.693402:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765824.693417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765824/real 1775765824] req@ffff9d81901ae680 x1862023066335744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765824.693441:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066335744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765824.693452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765824.693457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765824.693461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765824.693468:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765824.693487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765824.693496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765824/real 1775765824] req@ffff9d81901aed80 x1862023066335872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765824.693505:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066335872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765824.693512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765824.693514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765824.693517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066335872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765824.693521:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765824.693533:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765824.693540:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765824/real 1775765824] req@ffff9d81901afb80 x1862023066336000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765824.693548:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066336000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765824.693556:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765824.693566:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765824.693569:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765824.693576:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765824.693597:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765824.693609:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765824/real 1775765824] req@ffff9d81901ad180 x1862023066336128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765824.693625:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066336128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765824.693636:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765824.693640:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765824.693644:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765825.333078:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765825.716973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765826.357033:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765826.740939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765827.381056:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765827.765030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765828.405100:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765828.788994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765829.429033:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765829.812965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765829.812992:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765829.812998:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765829.813001:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765829.813008:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2661 00000100:00080000:3.0:1775765829.813017:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765829.813023:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765829.813056:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066336256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765829.813081:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765829.813084:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765829.813086:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765829.813089:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2661 00000100:00080000:3.0:1775765829.813091:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765829.813094:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765829.813102:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765829.813123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066336384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765829.813124:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066336256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765829.813137:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765829.813139:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765829.813140:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765829.813142:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2661 00000100:00080000:3.0:1775765829.813145:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765829.813147:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765829.813150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765829.813159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066336512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765829.813173:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765829.813176:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765829.813180:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775765829.813184:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765829.813186:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2661 00000100:00080000:3.0:1775765829.813190:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765829.813203:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765829.813207:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765829.813217:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066336640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765829.813223:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765829.813225:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765829.813226:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765829.813229:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2661 00000100:00080000:3.0:1775765829.813232:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765829.813233:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765829.813236:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765829.813244:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066336768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765829.813257:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765829/real 1775765829] req@ffff9d8188743800 x1862023066336256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765829.813277:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066336256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765829.813285:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765829.813288:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765829.813291:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066336256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765829.813309:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765829.813327:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765829.813335:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765829/real 1775765829] req@ffff9d8188742680 x1862023066336384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765829.813348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066336384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765829.813354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765829.813356:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765829.813358:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765829.813361:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765829.813370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765829.813376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765829/real 1775765829] req@ffff9d8188741880 x1862023066336512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765829.813383:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066336512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765829.813388:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765829.813390:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765829.813392:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765829.813395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765829.813405:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765829.813410:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765829/real 1775765829] req@ffff9d8188741c00 x1862023066336640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765829.813417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066336640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765829.813427:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765829.813429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765829.813431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765829.813434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765829.813443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765829.813448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765829/real 1775765829] req@ffff9d8188742a00 x1862023066336768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765829.813455:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066336768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765829.813460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765829.813462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765829.813463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066336768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765830.453088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765830.837020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765831.477038:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765831.861046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765832.501082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765832.885001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765833.525049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765833.909019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765834.549045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765834.932943:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765834.932953:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765834.932958:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775765834.932961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765834.932967:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2666 00000100:00080000:3.0:1775765834.932977:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765834.932985:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765834.933032:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066336896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765834.933069:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765834.933074:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765834.933076:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765834.933081:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2666 00000100:00080000:3.0:1775765834.933085:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765834.933090:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765834.933102:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765834.933140:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066337024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765834.933154:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066336896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765834.933159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765834.933168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765834.933171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765834.933176:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2666 00000100:00080000:3.0:1775765834.933181:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765834.933185:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765834.933191:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765834.933208:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066337152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765834.933225:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765834.933229:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765834.933230:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775765834.933237:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765834.933240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2666 00000100:00080000:3.0:1775765834.933247:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765834.933253:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765834.933261:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765834.933313:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066337280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765834.933316:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765834/real 1775765834] req@ffff9d8188740700 x1862023066336896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765834.933341:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765834.933345:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765834.933347:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775765834.933352:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066336896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765834.933354:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2666 00000100:00080000:3.0:1775765834.933362:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765834.933370:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765834.933378:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775765834.933382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765834.933389:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765834.933394:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066337408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765834.933395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066336896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765834.933426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765834.933447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765834.933457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765834/real 1775765834] req@ffff9d8188740380 x1862023066337024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765834.933467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066337024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765834.933473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765834.933476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765834.933479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765834.933483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765834.933504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765834.933511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765834/real 1775765834] req@ffff9d8188740000 x1862023066337152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765834.933520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066337152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765834.933527:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765834.933529:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765834.933532:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765834.933537:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765834.933557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765834.933570:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765834/real 1775765834] req@ffff9d8188742a00 x1862023066337280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765834.933579:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066337280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765834.933586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765834.933588:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765834.933590:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765834.933612:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765834.933630:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765834.933651:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765834/real 1775765834] req@ffff9d8188741c00 x1862023066337408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765834.933660:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066337408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765834.933667:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765834.933669:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765834.933671:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765835.573025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765835.956967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765836.597148:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765836.980950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765837.621063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765838.004980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765838.645077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765839.029007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765839.669097:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765840.052986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765840.053002:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765840.053013:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765840.053017:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765840.053028:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2671 00000100:00080000:0.0:1775765840.053039:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765840.053046:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765840.053085:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066337536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765840.053125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765840.053128:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765840.053130:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765840.053134:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2671 00000100:00080000:0.0:1775765840.053138:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765840.053143:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765840.053152:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765840.053161:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066337536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765840.053168:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066337664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765840.053180:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765840.053184:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765840.053186:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765840.053191:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2671 00000100:00080000:0.0:1775765840.053209:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765840.053214:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765840.053218:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775765840.053223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765840.053239:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066337792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765840.053250:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765840.053253:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765840.053256:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765840.053262:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2671 00000100:00100000:3.0:1775765840.053262:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765840/real 1775765840] req@ffff9d81901ad180 x1862023066337536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765840.053269:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765840.053277:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765840.053284:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765840.053288:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066337536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765840.053296:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775765840.053298:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066337920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765840.053304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765840.053311:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066337536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765840.053320:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765840.053323:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765840.053324:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765840.053328:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2671 00000100:00080000:0.0:1775765840.053332:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765840.053335:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337664:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775765840.053336:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765840.053345:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765840.053357:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066338048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765840.053365:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765840.053382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765840/real 1775765840] req@ffff9d81901afb80 x1862023066337664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765840.053391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066337664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765840.053397:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765840.053399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765840.053402:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765840.053405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765840.053415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765840.053421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765840/real 1775765840] req@ffff9d81901aed80 x1862023066337792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765840.053428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066337792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765840.053438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765840.053440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765840.053442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765840.053446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765840.053456:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765840.053460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765840/real 1775765840] req@ffff9d81901ae680 x1862023066337920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765840.053467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066337920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765840.053474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765840.053475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765840.053477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066337920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765840.053486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765840.053495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765840.053500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765840/real 1775765840] req@ffff9d81901af100 x1862023066338048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765840.053508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066338048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765840.053513:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765840.053514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765840.053517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765840.693033:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765841.076952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765841.717054:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765842.101000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765842.741068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765843.124982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765843.765239:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765844.148990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765844.789040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765845.173059:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765845.173080:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765845.173094:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765845.173102:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765845.173132:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2676 00000100:00080000:0.0:1775765845.173151:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765845.173166:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765845.173244:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066338176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765845.173304:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765845.173311:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765845.173315:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765845.173324:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2676 00000100:00080000:0.0:1775765845.173334:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765845.173343:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765845.173363:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765845.173378:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066338176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765845.173397:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066338304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765845.173430:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765845.173436:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765845.173439:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765845.173447:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2676 00000100:00080000:0.0:1775765845.173475:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765845.173481:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765845.173517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775765845.173522:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765845.173563:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066338432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765845.173578:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765845.173594:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765845.173598:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765845.173606:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2676 00000100:00080000:0.0:1775765845.173611:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765845.173614:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765845/real 1775765845] req@ffff9d81901af100 x1862023066338176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765845.173621:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765845.173637:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765845.173658:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066338560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765845.173662:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066338176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765845.173741:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765845.173742:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765845.173758:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:0.0:1775765845.173761:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775765845.173766:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066338176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765845.173769:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765845.173794:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2676 00000100:00080000:0.0:1775765845.173804:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765845.173814:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765845.173825:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338304:192.168.202.105@tcp:38:kworker.0 00000008:00100000:0.0:1775765845.173827:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775765845.173883:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765845.173917:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066338688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765845.173968:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765845/real 1775765845] req@ffff9d81901ae680 x1862023066338304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765845.173990:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066338304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765845.174002:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765845.174007:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765845.174011:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765845.174022:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765845.174066:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765845.174088:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765845/real 1775765845] req@ffff9d81901aed80 x1862023066338432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765845.174126:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066338432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765845.174136:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765845.174150:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765845.174153:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765845.174161:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765845.174189:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765845.174203:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765845/real 1775765845] req@ffff9d81901afb80 x1862023066338560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765845.174218:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066338560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765845.174228:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765845.174232:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765845.174236:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765845.174256:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765845.174279:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765845.174292:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765845/real 1775765845] req@ffff9d81901ad180 x1862023066338688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765845.174315:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066338688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765845.174326:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765845.174329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765845.174333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765845.813088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765846.197000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765846.837061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765847.221000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765847.861052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765848.245034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765848.885101:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765849.269014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765849.909187:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765850.293008:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765850.293013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765850.293024:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765850.293031:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765850.293046:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2681 00000100:00080000:0.0:1775765850.293072:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765850.293084:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765850.293139:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066338816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765850.293179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765850.293184:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765850.293186:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765850.293191:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2681 00000100:00080000:0.0:1775765850.293196:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765850.293201:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765850.293213:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765850.293230:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066338944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765850.293245:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765850.293250:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765850.293253:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765850.293262:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2681 00000100:00100000:3.0:1775765850.293264:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066338816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765850.293266:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765850.293277:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765850.293285:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765850.293303:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066339072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765850.293316:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765850.293321:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765850.293324:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765850.293328:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2681 00000100:00080000:0.0:1775765850.293344:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765850.293349:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765850.293357:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775765850.293358:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765850.293380:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066339200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765850.293394:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765850.293398:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765850.293400:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765850.293405:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2681 00000100:00080000:0.0:1775765850.293410:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765850.293416:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765850.293423:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765850.293427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765850/real 1775765850] req@ffff9d81901ad180 x1862023066338816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765850.293444:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066339328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765850.293513:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066338816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765850.293530:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765850.293537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765850.293542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066338816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765850.293552:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765850.293583:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765850.293599:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765850/real 1775765850] req@ffff9d81901afb80 x1862023066338944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765850.293623:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066338944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765850.293633:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765850.293637:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765850.293641:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066338944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765850.293669:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765850.293689:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765850.293700:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765850/real 1775765850] req@ffff9d81901aed80 x1862023066339072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765850.293714:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066339072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765850.293723:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765850.293727:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765850.293730:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765850.293737:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765850.293755:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765850.293765:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765850/real 1775765850] req@ffff9d81901ae680 x1862023066339200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765850.293779:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066339200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765850.293798:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765850.293802:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765850.293806:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765850.293813:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765850.293830:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765850.293840:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765850/real 1775765850] req@ffff9d81901af100 x1862023066339328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765850.293852:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066339328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765850.293873:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765850.293917:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765850.293922:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765850.933165:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765851.317037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765851.957141:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765852.341016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765852.981101:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765853.364943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765854.005255:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765854.388963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765855.029248:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765855.412956:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765855.412958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765855.412967:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765855.412972:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765855.412980:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2686 00000100:00080000:0.0:1775765855.412988:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765855.412994:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765855.413028:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066339456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765855.413051:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765855.413054:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765855.413055:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765855.413057:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2686 00000100:00080000:0.0:1775765855.413059:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765855.413061:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765855.413069:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765855.413076:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066339584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765855.413083:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765855.413086:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765855.413087:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765855.413090:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2686 00000100:00080000:0.0:1775765855.413092:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765855.413094:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765855.413097:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765855.413103:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066339712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765855.413109:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066339456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765855.413124:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765855.413126:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765855.413127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765855.413130:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2686 00000100:00080000:0.0:1775765855.413131:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765855.413133:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765855.413137:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765855.413144:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066339840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765855.413151:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765855.413153:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765855.413154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765855.413156:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2686 00000100:00080000:0.0:1775765855.413166:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765855.413169:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765855.413171:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775765855.413177:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765855.413180:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066339968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765855.413231:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765855/real 1775765855] req@ffff9d81901af100 x1862023066339456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765855.413252:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066339456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765855.413270:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765855.413274:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765855.413278:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066339456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765855.413286:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765855.413309:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765855.413323:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765855/real 1775765855] req@ffff9d81901ae680 x1862023066339584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765855.413337:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066339584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765855.413345:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765855.413349:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765855.413352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765855.413374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765855.413393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765855.413436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765855/real 1775765855] req@ffff9d81901aed80 x1862023066339712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765855.413449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066339712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765855.413459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765855.413462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765855.413466:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765855.413473:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765855.413491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765855.413501:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765855/real 1775765855] req@ffff9d81901afb80 x1862023066339840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765855.413514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066339840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765855.413523:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765855.413527:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765855.413530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765855.413537:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765855.413557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765855.413568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765855/real 1775765855] req@ffff9d81901ad180 x1862023066339968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765855.413581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066339968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765855.413589:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765855.413593:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765855.413596:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066339968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765856.053132:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765856.436955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765857.077124:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765857.461027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765858.101096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765858.484953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765859.125133:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765859.509012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765860.149025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765860.532945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765860.532975:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765860.532983:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765860.532987:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765860.532997:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2692 00000100:00080000:2.0:1775765860.533008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765860.533016:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765860.533051:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066340096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765860.533078:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765860.533082:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765860.533084:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765860.533089:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2692 00000100:00080000:2.0:1775765860.533093:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765860.533097:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765860.533108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765860.533123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066340224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765860.533128:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066340096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765860.533141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765860.533143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765860.533145:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765860.533149:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2692 00000100:00080000:2.0:1775765860.533153:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765860.533156:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765860.533161:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765860.533171:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066340352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765860.533177:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765860.533180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765860.533181:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765860.533183:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2692 00000100:00080000:2.0:1775765860.533186:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765860.533188:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765860.533191:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765860.533199:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066340480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765860.533205:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765860.533208:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765860.533209:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765860.533213:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2692 00000100:00080000:2.0:1775765860.533216:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765860.533219:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765860.533222:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765860.533229:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775765860.533234:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066340608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765860.533259:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765860.533274:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765860.533289:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765861.173100:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765861.557032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765862.197036:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765862.581039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765863.221087:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765863.605156:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775765863.605362:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775765863.605410:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775765863.605431:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765863.605480:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765863.605492:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765863.605501:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775765863.605511:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765863.605527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765860/real 1775765863] req@ffff9d81984a1500 x1862023066340224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765863.605560:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066340224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765863.605575:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765863.605581:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765863.605587:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765863.605597:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765860/real 1775765863] req@ffff9d81984a1180 x1862023066340352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765863.605612:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066340352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765863.605623:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765863.605628:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765863.605632:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765863.605662:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765860/real 1775765863] req@ffff9d81984a1f80 x1862023066340480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765863.605679:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066340480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765863.605690:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765863.605695:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765863.605699:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765863.605707:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765860/real 1775765863] req@ffff9d81984a2300 x1862023066340096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765863.605722:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066340096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765863.605732:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765863.605736:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765863.605741:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066340096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765863.605771:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765860/real 1775765863] req@ffff9d81984a2680 x1862023066340608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765863.605796:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066340608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765863.605821:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765863.605825:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765863.605829:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765864.245073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765864.629088:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765865.269053:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765865.653005:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765865.653018:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765865.653023:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765865.653035:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2697 00000100:00080000:2.0:1775765865.653049:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765865.653059:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765865.653097:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:2.0:1775765865.653123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066340736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765865.653173:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765865.653180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765865.653184:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765865.653191:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2697 00000100:00080000:2.0:1775765865.653199:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765865.653207:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765865.653223:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765865.653251:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066340736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765865.653253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066340864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765865.653285:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765865.653291:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765865.653294:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765865.653336:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2697 00000100:00080000:2.0:1775765865.653356:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765865.653365:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765865.653375:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775765865.653399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765865.653405:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066340992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765865.653429:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765865.653436:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765865.653439:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765865.653449:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2697 00000100:00080000:2.0:1775765865.653473:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765865.653482:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765865.653492:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765865.653494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765865/real 1775765865] req@ffff9d81984a0000 x1862023066340736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765865.653519:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066341120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765865.653538:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765865.653546:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765865.653548:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765865.653553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066340736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765865.653561:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2697 00000100:00080000:2.0:1775765865.653574:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765865.653595:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775765865.653596:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775765865.653606:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775765865.653619:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765865.653630:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066340736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765865.653635:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066341248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765865.653694:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765865.653770:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765865.653795:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765865/real 1775765865] req@ffff9d81984a2a00 x1862023066340864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765865.653819:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066340864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765865.653836:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765865.653843:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765865.653850:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765865.653862:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765865.653954:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765865.653977:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765865/real 1775765865] req@ffff9d81984a2d80 x1862023066340992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765865.654000:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066340992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765865.654017:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765865.654023:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765865.654029:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066340992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765865.654041:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765865.654079:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765865.654099:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765865/real 1775765865] req@ffff9d81984a0700 x1862023066341120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765865.654121:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066341120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765865.654137:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765865.654142:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765865.654147:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765865.654158:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765865.655071:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765865.655117:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765865/real 1775765865] req@ffff9d81984a3b80 x1862023066341248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765865.655136:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066341248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765865.655148:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765865.655152:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765865.655156:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765866.293079:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765866.677219:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765867.317078:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765867.701006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765868.341020:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765868.725004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765869.365011:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765869.748963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765870.389094:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765870.772966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765870.773011:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765870.773024:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765870.773031:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765870.773043:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2702 00000100:00080000:2.0:1775765870.773058:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765870.773066:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765870.773112:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066341376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765870.773152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765870.773157:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765870.773160:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765870.773165:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2702 00000100:00080000:2.0:1775765870.773170:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765870.773176:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765870.773189:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765870.773198:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066341376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765870.773208:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066341504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765870.773225:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765870.773230:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765870.773233:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765870.773239:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2702 00000100:00080000:2.0:1775765870.773277:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765870.773286:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765870.773294:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775765870.773306:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765870.773315:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066341632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765870.773333:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765870.773338:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765870.773340:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765870.773348:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2702 00000100:00080000:2.0:1775765870.773355:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765870.773361:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765870.773368:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765870.773393:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765870/real 1775765870] req@ffff9d81984a3480 x1862023066341376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765870.773395:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066341760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765870.773496:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765870.773502:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765870.773504:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765870.773510:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066341376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765870.773519:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2702 00000100:00080000:2.0:1775765870.773534:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765870.773551:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775765870.773555:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775765870.773559:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775765870.773563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765870.773577:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066341376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765870.773583:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066341888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765870.773610:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765870.773636:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765870.773646:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765870/real 1775765870] req@ffff9d81984a3b80 x1862023066341504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765870.773655:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066341504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765870.773663:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765870.773666:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765870.773668:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765870.773673:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765870.773684:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765870.773690:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765870/real 1775765870] req@ffff9d81984a0700 x1862023066341632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765870.773698:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066341632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765870.773703:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765870.773705:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765870.773707:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765870.773710:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765870.773723:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765870.773728:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765870/real 1775765870] req@ffff9d81984a2d80 x1862023066341760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765870.773735:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066341760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765870.773742:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765870.773744:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765870.773747:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765870.773751:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765870.773762:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765870.773768:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765870/real 1775765870] req@ffff9d81984a2a00 x1862023066341888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765870.773775:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066341888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765870.773781:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765870.773783:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765870.773784:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066341888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765871.412995:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765871.797067:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765872.437001:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765872.820994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765873.461089:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765873.844976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765874.485032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765874.868984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765875.509067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765875.892952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765875.893013:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765875.893020:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765875.893024:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765875.893033:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2707 00000100:00080000:0.0:1775765875.893044:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765875.893052:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765875.893098:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066342016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765875.893125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765875.893128:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765875.893129:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765875.893132:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2707 00000100:00080000:0.0:1775765875.893134:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765875.893144:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765875.893157:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765875.893172:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066342016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765875.893175:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066342144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765875.893190:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765875.893193:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765875.893196:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765875.893213:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2707 00000100:00080000:0.0:1775765875.893218:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765875.893224:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765875.893230:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765875.893243:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066342272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765875.893254:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765875.893258:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765875.893259:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765875.893263:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2707 00000100:00080000:0.0:1775765875.893267:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765875.893272:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765875.893274:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765875.893280:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765875.893293:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066342400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765875.893303:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765875.893307:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765875.893309:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765875.893314:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2707 00000100:00080000:0.0:1775765875.893318:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765875.893322:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765875.893327:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765875.893340:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066342528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765875.893361:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765875/real 1775765875] req@ffff9d81901ad180 x1862023066342016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765875.893385:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066342016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765875.893398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765875.893403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765875.893408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066342016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765875.893436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765875.893463:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765875.893477:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765875/real 1775765875] req@ffff9d81901afb80 x1862023066342144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765875.893492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066342144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765875.893501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765875.893505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765875.893508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765875.893516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765875.893536:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765875.893547:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765875/real 1775765875] req@ffff9d81901aed80 x1862023066342272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765875.893561:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066342272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765875.893570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765875.893574:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765875.893578:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765875.893584:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765875.893604:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765875.893615:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765875/real 1775765875] req@ffff9d81901ae680 x1862023066342400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765875.893636:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066342400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765875.893647:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765875.893651:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765875.893655:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765875.893661:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765875.893681:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765875.893692:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765875/real 1775765875] req@ffff9d81901af100 x1862023066342528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765875.893705:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066342528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765875.893715:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765875.893719:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765875.893724:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775765876.533058:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765876.916969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765877.557071:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765877.940954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765878.581074:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765878.964989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765879.605034:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765879.988950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765880.629054:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765881.012999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765881.013004:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765881.013010:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765881.013014:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765881.013023:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2712 00000100:00080000:0.0:1775765881.013033:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765881.013041:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765881.013081:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066342656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765881.013108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765881.013112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765881.013114:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765881.013118:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2712 00000100:00080000:0.0:1775765881.013122:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765881.013126:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765881.013136:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765881.013151:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066342784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765881.013175:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765881.013179:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765881.013180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765881.013186:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2712 00000100:00080000:0.0:1775765881.013190:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765881.013194:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765881.013198:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066342656:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775765881.013201:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765881.013214:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066342912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765881.013223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765881.013226:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765881.013227:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765881.013230:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2712 00000100:00080000:0.0:1775765881.013244:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765881.013247:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765881.013251:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765881.013254:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765881.013263:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066343040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765881.013270:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765881.013273:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765881.013274:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765881.013276:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2712 00000100:00080000:0.0:1775765881.013279:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765881.013281:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765881.013284:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765881.013293:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066343168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765881.013296:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765881/real 1775765881] req@ffff9d81901af100 x1862023066342656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765881.013327:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066342656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765881.013334:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765881.013337:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765881.013340:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066342656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765881.013345:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765881.013359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765881.013366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765881/real 1775765881] req@ffff9d81901ae680 x1862023066342784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765881.013373:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066342784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765881.013378:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765881.013380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765881.013382:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765881.013398:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765881.013408:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765881.013413:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765881/real 1775765881] req@ffff9d81901aed80 x1862023066342912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765881.013420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066342912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765881.013429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765881.013431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765881.013433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066342912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765881.013437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765881.013446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765881.013451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765881/real 1775765881] req@ffff9d81901afb80 x1862023066343040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765881.013458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066343040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765881.013464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765881.013466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765881.013467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765881.013471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765881.013480:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765881.013485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765881/real 1775765881] req@ffff9d81901ad180 x1862023066343168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765881.013492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066343168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765881.013497:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765881.013499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765881.013501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765881.653119:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765882.036993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765882.677051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765883.060996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765883.701053:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765884.085007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765884.725050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765885.108964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765885.749050:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765886.132946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765886.132958:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765886.132966:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765886.132970:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765886.132980:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2717 00000100:00080000:0.0:1775765886.132990:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765886.132996:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765886.133033:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066343296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765886.133064:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765886.133068:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765886.133070:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765886.133073:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2717 00000100:00080000:0.0:1775765886.133077:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765886.133080:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765886.133090:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765886.133120:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066343424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765886.133133:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066343296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765886.133135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765886.133143:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765886.133145:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765886.133150:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2717 00000100:00080000:0.0:1775765886.133156:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765886.133159:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765886.133165:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765886.133179:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066343552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765886.133190:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765886.133194:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765886.133196:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765886.133203:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2717 00000100:00080000:0.0:1775765886.133206:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765886.133209:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765886.133213:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765886.133219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765886.133231:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066343680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765886.133241:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765886.133244:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765886.133246:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765886.133251:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2717 00000100:00080000:0.0:1775765886.133255:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765886.133260:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765886.133261:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765886/real 1775765886] req@ffff9d81901ad180 x1862023066343296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775765886.133265:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765886.133280:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066343808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765886.133290:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066343296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765886.133314:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765886.133319:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765886.133324:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066343296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765886.133350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765886.133379:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765886.133394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765886/real 1775765886] req@ffff9d81901afb80 x1862023066343424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765886.133409:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066343424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765886.133418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765886.133422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765886.133425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765886.133432:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765886.133449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765886.133458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765886/real 1775765886] req@ffff9d81901aed80 x1862023066343552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765886.133466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066343552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765886.133472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765886.133474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765886.133476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765886.133480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765886.133490:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765886.133496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765886/real 1775765886] req@ffff9d81901ae680 x1862023066343680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765886.133503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066343680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765886.133509:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765886.133511:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765886.133513:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765886.133516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765886.133526:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765886.133531:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765886/real 1775765886] req@ffff9d81901af100 x1862023066343808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765886.133539:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066343808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765886.133544:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765886.133546:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765886.133548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066343808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765886.773091:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765887.156971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765887.797004:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765888.180968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775765888.821121:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765889.204951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765889.845043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765890.228984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765890.869015:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765891.252980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765891.252995:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765891.253002:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765891.253007:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765891.253017:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2722 00000100:00080000:2.0:1775765891.253027:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765891.253034:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765891.253092:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066343936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765891.253122:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765891.253126:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765891.253128:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765891.253132:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2722 00000100:00080000:2.0:1775765891.253136:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765891.253140:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765891.253150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765891.253165:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066343936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765891.253166:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066344064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765891.253182:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765891.253186:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765891.253188:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765891.253193:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2722 00000100:00080000:2.0:1775765891.253197:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765891.253201:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765891.253206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775765891.253224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765891.253228:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066344192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765891.253239:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765891.253243:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765891.253245:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765891.253250:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2722 00000100:00080000:2.0:1775765891.253254:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765891.253259:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765891.253264:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765891.253264:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765891/real 1775765891] req@ffff9d81984a0000 x1862023066343936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765891.253281:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066344320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765891.253285:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066343936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765891.253312:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765891.253315:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765891.253318:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775765891.253325:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775765891.253325:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066343936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765891.253330:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765891.253337:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2722 00000100:00080000:2.0:1775765891.253343:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765891.253347:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344064:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775765891.253349:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765891.253357:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765891.253377:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066344448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765891.253380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765891.253399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765891/real 1775765891] req@ffff9d81984a2680 x1862023066344064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765891.253421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066344064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765891.253430:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765891.253433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765891.253436:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765891.253441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765891.253458:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765891.253466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765891/real 1775765891] req@ffff9d81984a2300 x1862023066344192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765891.253473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066344192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765891.253479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765891.253481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765891.253483:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765891.253486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765891.253496:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765891.253502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765891/real 1775765891] req@ffff9d81984a1f80 x1862023066344320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765891.253509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066344320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765891.253519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765891.253521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765891.253523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765891.253533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765891.253542:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765891.253548:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765891/real 1775765891] req@ffff9d81984a1180 x1862023066344448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765891.253555:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066344448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765891.253560:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765891.253562:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765891.253564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765891.893065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765892.276965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765892.917069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765893.300967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765893.940981:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765894.324988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765894.965029:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765895.348961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765895.989102:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765896.372976:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765896.372980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765896.372988:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765896.372993:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765896.373003:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2727 00000100:00080000:2.0:1775765896.373033:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765896.373041:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765896.373098:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066344576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765896.373126:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765896.373129:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765896.373131:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765896.373134:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2727 00000100:00080000:2.0:1775765896.373137:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765896.373140:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765896.373148:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765896.373160:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066344704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765896.373168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765896.373172:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765896.373174:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765896.373177:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2727 00000100:00080000:2.0:1775765896.373181:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765896.373182:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066344576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765896.373186:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765896.373193:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765896.373201:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066344832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765896.373208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765896.373210:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765896.373212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765896.373214:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2727 00000100:00080000:2.0:1775765896.373216:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765896.373218:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765896.373222:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765896.373228:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066344960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765896.373235:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765896.373237:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765896.373238:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765896.373240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2727 00000800:00000100:3.0:1775765896.373244:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765896.373245:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765896.373250:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765896.373253:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765896.373262:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066345088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765896.373288:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765896/real 1775765896] req@ffff9d81984a1500 x1862023066344576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765896.373303:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066344576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765896.373311:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765896.373314:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765896.373317:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066344576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765896.373322:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765896.373336:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765896.373343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765896/real 1775765896] req@ffff9d81984a1880 x1862023066344704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765896.373351:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066344704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765896.373356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765896.373358:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765896.373360:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765896.373376:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765896.373386:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765896.373391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765896/real 1775765896] req@ffff9d81984a3800 x1862023066344832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765896.373398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066344832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765896.373403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765896.373406:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765896.373408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765896.373412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765896.373422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765896.373427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765896/real 1775765896] req@ffff9d81984a3100 x1862023066344960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765896.373434:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066344960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765896.373440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765896.373442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765896.373444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066344960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765896.373447:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765896.373465:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765896.373472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765896/real 1775765896] req@ffff9d81984a0e00 x1862023066345088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765896.373479:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066345088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765896.373485:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765896.373487:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765896.373489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765897.013066:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765897.396981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765898.037143:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765898.420979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765899.061021:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765899.444948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765900.085086:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765900.468958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765901.109065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765901.492933:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775765901.492941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765901.492950:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765901.492955:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765901.492979:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2733 00000100:00080000:2.0:1775765901.492989:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765901.492997:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765901.493030:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066345216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765901.493052:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765901.493054:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765901.493056:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765901.493059:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2733 00000100:00080000:2.0:1775765901.493061:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765901.493064:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765901.493072:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765901.493084:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066345344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765901.493089:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066345216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765901.493105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765901.493109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765901.493111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765901.493116:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2733 00000100:00080000:2.0:1775765901.493121:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765901.493125:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765901.493131:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765901.493144:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066345472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765901.493149:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765901.493156:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765901.493159:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765901.493161:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765901.493165:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2733 00000100:00080000:2.0:1775765901.493168:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765901.493171:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765901.493174:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765901.493183:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066345600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765901.493189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765901.493191:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765901/real 1775765901] req@ffff9d81984a0a80 x1862023066345216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765901.493192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765901.493195:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765901.493202:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2733 00000100:00080000:2.0:1775765901.493206:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765901.493209:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765901.493214:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765901.493214:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066345216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765901.493226:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765901.493231:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:2.0:1775765901.493232:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066345728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765901.493236:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066345216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765901.493259:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765901.493284:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765901.493293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765901/real 1775765901] req@ffff9d81984a0e00 x1862023066345344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765901.493302:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066345344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765901.493307:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765901.493309:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765901.493312:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765901.493315:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765901.493329:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765901.493334:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765901/real 1775765901] req@ffff9d81984a3100 x1862023066345472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765901.493342:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066345472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765901.493348:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765901.493349:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765901.493351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765901.493355:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765901.493366:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765901.493371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765901/real 1775765901] req@ffff9d81984a3800 x1862023066345600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765901.493396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066345600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765901.493412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765901.493414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765901.493416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765901.493428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765901.493459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765901.493466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765901/real 1775765901] req@ffff9d81984a1880 x1862023066345728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765901.493478:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066345728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765901.493483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765901.493485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765901.493487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765902.133146:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765902.517009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765903.157113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765903.541039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765904.181100:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765904.564975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765905.205135:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765905.589003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765906.229060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765906.612996:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765906.613005:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765906.613009:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765906.613017:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2738 00000100:00080000:2.0:1775765906.613026:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765906.613032:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765906.613063:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066345856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765906.613087:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765906.613091:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765906.613093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765906.613097:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2738 00000100:00080000:2.0:1775765906.613100:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765906.613113:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765906.613123:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765906.613128:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066345856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765906.613138:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066345984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765906.613150:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765906.613153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765906.613155:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765906.613159:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2738 00000100:00080000:2.0:1775765906.613178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765906.613183:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765906.613187:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775765906.613188:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765906.613201:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066346112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775765906.613210:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765906.613211:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765906.613221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765906.613222:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765906.613228:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2738 00000100:00080000:2.0:1775765906.613232:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765906.613237:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765906.613242:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765906.613244:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765906/real 1775765906] req@ffff9d81984a1500 x1862023066345856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765906.613252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066346240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765906.613261:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765906.613264:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775765906.613264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066345856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765906.613266:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765906.613270:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2738 00000100:00080000:2.0:1775765906.613273:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765906.613274:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775765906.613276:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775765906.613278:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:2.0:1775765906.613281:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765906.613283:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066345856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765906.613289:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066346368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765906.613305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765906.613327:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765906.613340:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765906/real 1775765906] req@ffff9d81984a1180 x1862023066345984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765906.613351:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066345984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765906.613357:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765906.613359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765906.613362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066345984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765906.613366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765906.613380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765906.613386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765906/real 1775765906] req@ffff9d81984a1f80 x1862023066346112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765906.613394:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066346112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765906.613399:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765906.613402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765906.613403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765906.613407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765906.613417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765906.613422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765906/real 1775765906] req@ffff9d81984a2300 x1862023066346240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765906.613429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066346240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765906.613439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765906.613441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765906.613443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765906.613446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765906.613455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765906.613459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765906/real 1775765906] req@ffff9d81984a2680 x1862023066346368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765906.613466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066346368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765906.613471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765906.613473:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765906.613475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765907.253066:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765907.637045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765908.277134:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765908.660984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765909.301042:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765909.684994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765910.325072:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765910.709018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765911.349112:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765911.732983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765911.733005:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765911.733014:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765911.733019:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765911.733030:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2743 00000100:00080000:0.0:1775765911.733042:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765911.733049:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765911.733099:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066346496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765911.733128:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765911.733136:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765911.733138:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765911.733141:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2743 00000100:00080000:0.0:1775765911.733144:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765911.733148:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765911.733158:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765911.733167:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066346624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765911.733176:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765911.733181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765911.733183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765911.733184:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066346496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765911.733189:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2743 00000100:00080000:0.0:1775765911.733193:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765911.733196:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765911.733199:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765911.733210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066346752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765911.733216:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765911.733218:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765911.733220:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765911.733222:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2743 00000100:00080000:0.0:1775765911.733225:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765911.733228:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765911.733231:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765911.733256:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066346880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765911.733262:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765911.733264:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765911.733266:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765911.733269:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2743 00000100:00080000:0.0:1775765911.733273:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765911.733275:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765911.733279:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765911.733287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066347008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765911.733295:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765911.733353:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765911/real 1775765911] req@ffff9d81901af100 x1862023066346496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765911.733372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066346496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765911.733384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765911.733388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765911.733391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066346496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765911.733415:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765911.733435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765911.733445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765911/real 1775765911] req@ffff9d81901ae680 x1862023066346624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765911.733454:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066346624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765911.733477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765911.733479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765911.733481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765911.733486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765911.733498:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765911.733504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765911/real 1775765911] req@ffff9d81901aed80 x1862023066346752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765911.733511:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066346752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765911.733517:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765911.733519:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765911.733522:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765911.733526:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765911.733538:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765911.733545:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765911/real 1775765911] req@ffff9d81901afb80 x1862023066346880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765911.733558:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066346880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765911.733568:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765911.733572:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765911.733576:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066346880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765911.733582:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765911.733621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765911.733634:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765911/real 1775765911] req@ffff9d81901ad180 x1862023066347008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765911.733644:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066347008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765911.733651:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765911.733654:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765911.733656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765912.373154:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765912.757074:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765913.397037:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765913.781007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765914.421059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765914.805030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765915.445046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775765915.829025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765916.469141:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775765916.853039:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765916.853050:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765916.853056:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765916.853069:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2748 00000100:00080000:1.0:1775765916.853083:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765916.853092:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765916.853140:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066347136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765916.853153:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765916.853174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765916.853180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765916.853184:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765916.853190:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2748 00000100:00080000:1.0:1775765916.853195:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765916.853199:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066347136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765916.853204:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765916.853218:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765916.853234:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066347264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765916.853246:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765916.853250:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765916.853252:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765916.853256:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2748 00000100:00080000:1.0:1775765916.853279:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765916.853282:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765916.853287:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775765916.853294:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765916.853304:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066347392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765916.853315:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765916.853318:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765916.853320:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765916.853323:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2748 00000100:00080000:1.0:1775765916.853326:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765916.853331:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765916.853336:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765916.853352:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066347520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765916.853356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765916/real 1775765916] req@ffff9d8190551500 x1862023066347136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765916.853419:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765916.853423:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765916.853425:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775765916.853430:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066347136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775765916.853436:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2748 00000100:00080000:1.0:1775765916.853445:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765916.853455:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775765916.853458:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775765916.853461:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775765916.853468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775765916.853477:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066347648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765916.853488:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066347136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765916.853528:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765916.853558:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765916.853570:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765916/real 1775765916] req@ffff9d8190553480 x1862023066347264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765916.853584:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066347264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765916.853593:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765916.853597:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765916.853600:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765916.853606:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765916.853625:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765916.853643:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765916/real 1775765916] req@ffff9d8190553100 x1862023066347392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765916.853655:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066347392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765916.853664:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765916.853666:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765916.853669:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765916.853676:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765916.853692:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765916.853711:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765916/real 1775765916] req@ffff9d8190550e00 x1862023066347520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765916.853723:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066347520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765916.853732:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765916.853735:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765916.853738:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765916.853743:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765916.853758:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765916.853766:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765916/real 1775765916] req@ffff9d8190551880 x1862023066347648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765916.853785:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066347648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765916.853794:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765916.853797:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765916.853800:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765917.493190:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765917.876994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765918.517106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765918.901042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765919.541061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765919.924986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765920.565020:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765920.948974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765921.589027:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765921.973028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765921.973046:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765921.973056:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765921.973063:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765921.973078:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2753 00000100:00080000:2.0:1775765921.973095:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765921.973109:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765921.973169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066347776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765921.973215:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765921.973221:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765921.973224:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765921.973229:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2753 00000100:00080000:2.0:1775765921.973235:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765921.973242:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765921.973256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765921.973280:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066347776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765921.973283:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066347904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765921.973313:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765921.973322:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765921.973325:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765921.973333:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2753 00000100:00080000:2.0:1775765921.973342:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765921.973349:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765921.973360:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765921.973382:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066348032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765921.973397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765921.973417:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765921.973421:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765921.973425:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765921.973432:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2753 00000100:00080000:2.0:1775765921.973438:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765921.973442:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765921.973450:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765921.973467:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066348160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765921.973481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765921/real 1775765921] req@ffff9d81984a0000 x1862023066347776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765921.973484:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765921.973498:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765921.973503:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765921.973518:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2753 00000100:00080000:2.0:1775765921.973528:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765921.973533:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765921.973540:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765921.973544:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066347776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765921.973559:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066348288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765921.973566:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765921.973588:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765921.973598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066347776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765921.973640:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765921.973691:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765921.973719:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765921/real 1775765921] req@ffff9d81984a2a00 x1862023066347904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765921.973748:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066347904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765921.973795:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765921.973803:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765921.973811:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066347904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765921.973824:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765921.973870:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765921.973944:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765921/real 1775765921] req@ffff9d81984a2d80 x1862023066348032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765921.973972:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066348032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765921.973989:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765921.973997:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765921.974003:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765921.974016:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765921.974060:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765921.974101:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765921/real 1775765921] req@ffff9d81984a0700 x1862023066348160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765921.974129:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066348160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765921.974172:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765921.974179:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765921.974186:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765921.974199:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765921.974240:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765921.974263:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765921/real 1775765921] req@ffff9d81984a3b80 x1862023066348288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765921.974287:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066348288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765921.974304:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765921.974310:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765921.974317:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765922.613061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765922.996987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765923.637074:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765924.020969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765924.661099:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765925.045021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765925.685121:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765926.069060:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765926.709145:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765927.092991:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775765927.092996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765927.093001:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765927.093005:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765927.093013:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2758 00000100:00080000:0.0:1775765927.093021:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765927.093028:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765927.093063:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066348416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765927.093086:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765927.093089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765927.093091:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765927.093095:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2758 00000100:00080000:0.0:1775765927.093098:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765927.093102:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765927.093112:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765927.093125:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066348544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765927.093136:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066348416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765927.093141:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765927.093144:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765927.093146:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765927.093150:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2758 00000100:00080000:0.0:1775765927.093153:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765927.093156:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765927.093161:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765927.093169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066348672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765927.093177:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765927.093180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765927.093182:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765927.093196:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2758 00000100:00080000:0.0:1775765927.093199:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765927.093202:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765927.093207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765927.093215:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066348800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765927.093225:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765927.093228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765927.093230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765927.093243:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2758 00000100:00080000:0.0:1775765927.093263:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775765927.093265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348544:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775765927.093267:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765927.093272:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765927.093284:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066348928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765927.093296:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765927.093317:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765927.093358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765927.733022:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765928.117041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765928.757029:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765929.141105:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765929.781043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765930.165111:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775765930.165235:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775765930.165277:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775765930.165293:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765930.165328:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765930.165336:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765930.165343:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765930.165358:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775765930.165386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765927/real 1775765930] req@ffff9d81901af100 x1862023066348928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765930.165414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066348928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765930.165425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765930.165430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765930.165434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348928:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765930.165441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765927/real 1775765930] req@ffff9d81901afb80 x1862023066348544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765930.165452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066348544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765930.165459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765930.165463:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765930.165466:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765930.165486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765927/real 1775765930] req@ffff9d81901aed80 x1862023066348672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765930.165494:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066348672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765930.165500:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765930.165502:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765930.165504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765930.165508:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765927/real 1775765930] req@ffff9d81901ae680 x1862023066348800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765930.165515:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066348800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765930.165520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765930.165522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765930.165524:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066348800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765930.165528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765927/real 1775765930] req@ffff9d81901ad180 x1862023066348416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765930.165535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066348416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765930.165539:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765930.165541:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765930.165543:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066348416:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775765930.805003:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765931.189018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765931.829049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775765932.212973:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765932.212983:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765932.212987:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765932.212997:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2763 00000100:00080000:0.0:1775765932.213008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765932.213016:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765932.213028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:0.0:1775765932.213053:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066349056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765932.213098:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765932.213113:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765932.213116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765932.213121:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2763 00000100:00080000:0.0:1775765932.213127:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765932.213132:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765932.213146:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765932.213166:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066349184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765932.213180:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066349056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765932.213190:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765932.213199:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765932.213202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765932.213210:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2763 00000100:00080000:0.0:1775765932.213217:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765932.213223:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765932.213231:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775765932.213260:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765932.213265:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066349312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765932.213279:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765932.213284:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765932.213287:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765932.213294:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2763 00000100:00080000:0.0:1775765932.213301:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765932.213307:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765932.213313:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765932/real 1775765932] req@ffff9d81901ad180 x1862023066349056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775765932.213318:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765932.213349:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066349440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765932.213349:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066349056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765932.213401:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765932.213403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765932.213411:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765932.213417:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765932.213417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:0.0:1775765932.213429:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2763 00000100:00100000:3.0:1775765932.213433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066349056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765932.213435:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765932.213446:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765932.213454:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765932.213472:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066349568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765932.213483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765932.213526:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765932.213549:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765932/real 1775765932] req@ffff9d81901ae680 x1862023066349184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765932.213569:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066349184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765932.213581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765932.213596:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765932.213603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765932.213613:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765932.213643:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765932.213659:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765932/real 1775765932] req@ffff9d81901aed80 x1862023066349312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765932.213715:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066349312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765932.213736:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765932.213741:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765932.213745:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765932.213753:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765932.213971:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765932.213988:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765932/real 1775765932] req@ffff9d81901afb80 x1862023066349440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765932.214002:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066349440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765932.214012:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765932.214016:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765932.214019:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765932.214039:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765932.214063:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765932.214074:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765932/real 1775765932] req@ffff9d81901af100 x1862023066349568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765932.214087:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066349568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765932.214096:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765932.214099:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765932.214103:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765932.853039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765933.237038:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765933.877043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765934.260951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765934.901122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765935.285001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765935.925055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765936.308935:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765936.949040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765937.332963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765937.333013:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765937.333019:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765937.333023:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765937.333031:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2768 00000100:00080000:1.0:1775765937.333040:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765937.333046:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765937.333085:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066349696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765937.333117:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765937.333121:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765937.333123:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765937.333127:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2768 00000100:00080000:1.0:1775765937.333131:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765937.333135:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765937.333143:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765937.333157:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066349824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765937.333158:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066349696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775765937.333173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765937.333176:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765937.333177:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765937.333180:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2768 00000100:00080000:1.0:1775765937.333183:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765937.333185:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765937.333188:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765937.333197:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066349952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765937.333203:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765937.333205:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765937.333206:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765937.333209:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2768 00000100:00080000:1.0:1775765937.333213:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765937.333215:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765937.333219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765937.333223:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775765937.333231:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066350080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765937.333238:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765937.333241:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765937.333242:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765937.333244:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2768 00000100:00080000:1.0:1775765937.333246:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765937.333249:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765937.333252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765937.333258:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066350208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765937.333277:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765937/real 1775765937] req@ffff9d8190551880 x1862023066349696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765937.333294:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066349696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765937.333302:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765937.333305:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765937.333308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066349696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765937.333325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765937.333341:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765937.333349:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765937/real 1775765937] req@ffff9d8190550e00 x1862023066349824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765937.333357:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066349824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765937.333362:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765937.333364:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765937.333367:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765937.333371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765937.333381:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765937.333387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765937/real 1775765937] req@ffff9d8190553100 x1862023066349952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765937.333394:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066349952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765937.333398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765937.333400:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765937.333402:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066349952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765937.333406:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765937.333416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765937.333421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765937/real 1775765937] req@ffff9d8190553480 x1862023066350080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765937.333428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066350080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765937.333433:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765937.333435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765937.333437:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765937.333440:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765937.333449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765937.333455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765937/real 1775765937] req@ffff9d8190551500 x1862023066350208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765937.333462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066350208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765937.333467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765937.333469:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765937.333470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765937.973193:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765938.356978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765938.997022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765939.380968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765940.021044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765940.404939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765941.045035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765941.428960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765942.069035:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775765942.452985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765942.452994:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765942.453000:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765942.453004:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765942.453013:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2773 00000100:00080000:0.0:1775765942.453022:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765942.453028:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765942.453063:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066350336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765942.453093:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765942.453096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765942.453097:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765942.453100:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2773 00000100:00080000:0.0:1775765942.453102:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765942.453105:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765942.453113:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765942.453123:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066350464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765942.453131:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765942.453132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066350336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775765942.453134:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765942.453137:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765942.453144:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2773 00000100:00080000:0.0:1775765942.453147:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765942.453149:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765942.453152:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765942.453160:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066350592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765942.453166:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765942.453168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765942.453169:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765942.453172:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2773 00000100:00080000:0.0:1775765942.453174:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765942.453176:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765942.453179:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765942.453185:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066350720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765942.453191:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765942.453194:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765942.453195:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765942.453197:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2773 00000100:00080000:0.0:1775765942.453200:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765942.453204:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775765942.453206:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765942.453210:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765942.453217:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066350848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765942.453256:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765942/real 1775765942] req@ffff9d81901af100 x1862023066350336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765942.453272:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066350336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765942.453281:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765942.453284:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765942.453288:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066350336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765942.453305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765942.453323:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765942.453331:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765942/real 1775765942] req@ffff9d81901afb80 x1862023066350464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765942.453340:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066350464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765942.453345:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765942.453347:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765942.453350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765942.453353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765942.453362:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765942.453367:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765942/real 1775765942] req@ffff9d81901aed80 x1862023066350592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765942.453374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066350592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765942.453379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765942.453381:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765942.453383:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765942.453386:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765942.453396:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765942.453401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765942/real 1775765942] req@ffff9d81901ae680 x1862023066350720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765942.453413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066350720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765942.453419:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765942.453421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765942.453422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765942.453426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765942.453434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765942.453439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765942/real 1775765942] req@ffff9d81901ad180 x1862023066350848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765942.453446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066350848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765942.453452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765942.453453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765942.453455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066350848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765943.093082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765943.477035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765944.117111:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765944.500994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765945.141078:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765945.524965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765946.165101:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765946.549006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765947.189022:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765947.572936:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775765947.572943:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765947.572948:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775765947.572953:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765947.572968:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2779 00000100:00080000:0.0:1775765947.572977:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765947.572983:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775765947.573016:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066350976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765947.573038:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765947.573040:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775765947.573042:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765947.573044:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2779 00000100:00080000:0.0:1775765947.573047:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765947.573049:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765947.573056:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765947.573063:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066351104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765947.573077:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765947.573084:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775765947.573086:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765947.573091:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2779 00000100:00080000:0.0:1775765947.573096:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765947.573100:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775765947.573106:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765947.573109:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066350976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765947.573122:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066351232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765947.573133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765947.573137:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775765947.573139:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765947.573144:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2779 00000100:00080000:0.0:1775765947.573156:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765947.573159:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775765947.573164:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775765947.573165:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765947.573181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066351360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765947.573192:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775765947.573196:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775765947.573198:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775765947.573202:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2779 00000100:00080000:0.0:1775765947.573206:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775765947.573211:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765947.573216:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765947/real 1775765947] req@ffff9d81901ad180 x1862023066350976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775765947.573217:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775765947.573254:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066351488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765947.573269:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066350976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765947.573279:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765947.573283:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765947.573287:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066350976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765947.573293:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765947.573312:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765947.573325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765947/real 1775765947] req@ffff9d81901ae680 x1862023066351104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765947.573339:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066351104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765947.573347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765947.573351:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765947.573354:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765947.573377:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765947.573393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765947.573402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765947/real 1775765947] req@ffff9d81901aed80 x1862023066351232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765947.573410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066351232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765947.573438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765947.573440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765947.573442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765947.573446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765947.573457:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765947.573463:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765947/real 1775765947] req@ffff9d81901afb80 x1862023066351360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765947.573470:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066351360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765947.573476:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765947.573477:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765947.573479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765947.573483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765947.573492:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765947.573497:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765947/real 1775765947] req@ffff9d81901af100 x1862023066351488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765947.573504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066351488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765947.573510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765947.573511:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765947.573514:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765948.213033:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765948.597010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765949.237156:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765949.621031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765950.261000:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765950.644976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765951.285082:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775765951.668988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765952.309124:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765952.692976:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765952.692983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765952.692984:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765952.692989:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765952.692998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2784 00000100:00080000:2.0:1775765952.693006:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765952.693013:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765952.693046:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066351616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765952.693068:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765952.693075:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765952.693077:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765952.693081:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2784 00000100:00080000:2.0:1775765952.693084:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765952.693087:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765952.693095:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765952.693106:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066351616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765952.693112:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066351744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765952.693121:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765952.693123:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765952.693125:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765952.693128:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2784 00000100:00080000:2.0:1775765952.693131:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765952.693133:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765952.693136:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765952.693146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066351872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765952.693152:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765952.693154:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765952.693155:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765952.693158:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2784 00000100:00080000:2.0:1775765952.693161:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765952.693163:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765952.693165:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765952.693171:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765952.693175:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066352000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765952.693182:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765952.693184:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765952.693185:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765952.693187:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2784 00000100:00080000:2.0:1775765952.693189:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765952.693191:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765952.693194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765952.693200:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066352128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765952.693232:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765952/real 1775765952] req@ffff9d81984a3480 x1862023066351616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765952.693248:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066351616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765952.693257:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765952.693260:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765952.693263:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066351616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765952.693281:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765952.693298:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765952.693306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765952/real 1775765952] req@ffff9d81984a3b80 x1862023066351744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765952.693315:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066351744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765952.693320:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765952.693323:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765952.693325:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765952.693329:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765952.693340:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765952.693345:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765952/real 1775765952] req@ffff9d81984a0700 x1862023066351872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765952.693352:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066351872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765952.693358:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765952.693359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765952.693361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066351872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765952.693365:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765952.693375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765952.693380:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765952/real 1775765952] req@ffff9d81984a2d80 x1862023066352000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765952.693387:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066352000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765952.693393:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765952.693395:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765952.693397:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765952.693400:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765952.693409:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765952.693415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765952/real 1775765952] req@ffff9d81984a2a00 x1862023066352128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765952.693422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066352128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765952.693428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765952.693430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765952.693431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765953.333053:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765953.716940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765954.357083:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765954.740956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765955.381082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765955.764990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765956.405052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765956.789040:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765957.429004:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765957.812956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775765957.812962:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765957.812970:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775765957.812975:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765957.812984:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2789 00000100:00080000:1.0:1775765957.812996:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765957.813013:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775765957.813068:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066352256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765957.813097:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765957.813103:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775765957.813105:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765957.813110:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2789 00000100:00080000:1.0:1775765957.813114:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765957.813119:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765957.813131:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765957.813138:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066352256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775765957.813148:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066352384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765957.813160:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765957.813164:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775765957.813166:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765957.813171:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2789 00000100:00080000:1.0:1775765957.813175:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765957.813179:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765957.813185:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775765957.813198:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066352512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765957.813205:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765957.813208:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775765957.813210:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765957.813213:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2789 00000100:00080000:1.0:1775765957.813217:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775765957.813221:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775765957.813222:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765957.813228:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775765957.813237:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066352640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775765957.813244:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775765957.813247:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775765957.813248:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775765957.813250:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2789 00000100:00080000:1.0:1775765957.813253:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775765957.813255:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775765957.813259:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775765957.813270:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066352768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765957.813276:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765957/real 1775765957] req@ffff9d8190551500 x1862023066352256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765957.813295:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066352256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765957.813303:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765957.813307:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765957.813310:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066352256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765957.813331:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765957.813348:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765957.813357:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765957/real 1775765957] req@ffff9d8190553480 x1862023066352384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765957.813365:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066352384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765957.813371:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765957.813373:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765957.813376:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765957.813379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765957.813389:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765957.813394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765957/real 1775765957] req@ffff9d8190553100 x1862023066352512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765957.813402:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066352512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765957.813407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765957.813409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765957.813411:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765957.813414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765957.813444:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765957.813451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765957/real 1775765957] req@ffff9d8190550e00 x1862023066352640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765957.813459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066352640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765957.813466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765957.813468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765957.813470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765957.813474:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765957.813486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765957.813491:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765957/real 1775765957] req@ffff9d8190551880 x1862023066352768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765957.813498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066352768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765957.813504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765957.813505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765957.813507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066352768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765958.453050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765958.836982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765959.477008:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765959.860954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765960.501098:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765960.884969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765961.525134:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765961.908968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765962.549044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765962.932969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765962.933018:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765962.933029:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765962.933035:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765962.933048:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2794 00000100:00080000:2.0:1775765962.933063:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765962.933072:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765962.933124:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066352896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765962.933155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765962.933158:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765962.933160:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765962.933163:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2794 00000100:00080000:2.0:1775765962.933166:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765962.933170:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765962.933179:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765962.933189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066353024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765962.933197:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765962.933200:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765962.933201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765962.933205:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2794 00000100:00080000:2.0:1775765962.933207:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765962.933209:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765962.933214:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765962.933215:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066352896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765962.933224:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066353152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765962.933232:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765962.933234:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765962.933235:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765962.933237:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2794 00000100:00080000:2.0:1775765962.933240:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765962.933242:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765962.933246:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765962.933255:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066353280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765962.933262:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765962.933265:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765962.933266:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765962.933269:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2794 00000100:00080000:2.0:1775765962.933278:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765962.933281:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765962.933284:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775765962.933291:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765962.933295:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066353408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765962.933344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765962/real 1775765962] req@ffff9d81984a0000 x1862023066352896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765962.933360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066352896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765962.933387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765962.933392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765962.933397:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066352896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765962.933404:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765962.933423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765962.933433:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765962/real 1775765962] req@ffff9d81984a2680 x1862023066353024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765962.933442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066353024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765962.933448:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765962.933450:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765962.933452:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765962.933474:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765962.933485:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765962.933491:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765962/real 1775765962] req@ffff9d81984a2300 x1862023066353152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765962.933498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066353152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765962.933504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765962.933506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765962.933508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765962.933512:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765962.933523:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765962.933528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765962/real 1775765962] req@ffff9d81984a1f80 x1862023066353280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765962.933535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066353280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765962.933543:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765962.933546:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765962.933549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765962.933555:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765962.933574:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765962.933585:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765962/real 1775765962] req@ffff9d81984a1180 x1862023066353408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765962.933598:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066353408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765962.933608:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765962.933611:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765962.933613:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765963.572999:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765963.956977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765964.597028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765964.980954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765965.621039:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765966.004993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765966.645042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765967.028938:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765967.669037:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765968.052968:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765968.052969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765968.052977:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765968.052981:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765968.052990:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2799 00000100:00080000:2.0:1775765968.052999:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765968.053005:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765968.053038:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066353536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765968.053061:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765968.053064:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765968.053066:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765968.053069:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2799 00000100:00080000:2.0:1775765968.053072:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765968.053076:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765968.053085:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765968.053098:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066353664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765968.053109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765968.053113:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765968.053116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765968.053121:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2799 00000100:00080000:2.0:1775765968.053125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765968.053129:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765968.053134:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765968.053136:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066353536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765968.053150:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066353792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765968.053163:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765968.053167:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765968.053169:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765968.053173:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2799 00000100:00080000:2.0:1775765968.053184:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765968.053187:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765968.053191:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765968.053192:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765968.053210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066353920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765968.053217:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765968.053220:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765968.053221:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765968.053225:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2799 00000100:00080000:2.0:1775765968.053228:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765968.053230:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765968.053233:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765968.053236:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765968/real 1775765968] req@ffff9d81984a1500 x1862023066353536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775765968.053243:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066354048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765968.053255:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066353536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765968.053264:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765968.053268:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765968.053271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066353536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765968.053276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765968.053290:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765968.053297:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765968/real 1775765968] req@ffff9d81984a1880 x1862023066353664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765968.053305:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066353664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765968.053310:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765968.053312:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765968.053314:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765968.053359:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765968.053375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765968.053382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765968/real 1775765968] req@ffff9d81984a3800 x1862023066353792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765968.053390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066353792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765968.053395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765968.053397:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765968.053399:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765968.053403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765968.053413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765968.053419:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765968/real 1775765968] req@ffff9d81984a3100 x1862023066353920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765968.053426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066353920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765968.053431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765968.053433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765968.053435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066353920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765968.053448:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765968.053459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765968.053464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765968/real 1775765968] req@ffff9d81984a0e00 x1862023066354048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765968.053476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066354048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765968.053481:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765968.053483:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765968.053485:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765968.693057:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765969.076972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765969.717063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765970.100991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765970.741004:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765971.124969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765971.765092:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765972.149039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765972.789044:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765973.173009:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765973.173011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765973.173031:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765973.173036:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765973.173045:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2804 00000100:00080000:2.0:1775765973.173065:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765973.173074:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765973.173114:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066354176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765973.173157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765973.173161:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765973.173163:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765973.173167:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2804 00000100:00080000:2.0:1775765973.173171:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765973.173176:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765973.173186:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765973.173202:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066354304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765973.173223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765973.173225:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066354176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765973.173236:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765973.173238:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765973.173243:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2804 00000100:00080000:2.0:1775765973.173248:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765973.173253:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765973.173259:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765973.173302:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066354432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775765973.173314:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765973.173315:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765973.173320:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765973.173322:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765973.173329:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2804 00000100:00080000:2.0:1775765973.173334:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765973.173348:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765973.173354:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765973.173367:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066354560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765973.173378:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775765973.173380:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765973/real 1775765973] req@ffff9d81984a0a80 x1862023066354176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765973.173384:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765973.173389:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765973.173396:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2804 00000100:00080000:2.0:1775765973.173403:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765973.173408:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765973.173424:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765973.173428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066354176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765973.173439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765973.173444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:2.0:1775765973.173450:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066354688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765973.173451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066354176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765973.173478:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765973.173511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765973.173523:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765973/real 1775765973] req@ffff9d81984a0e00 x1862023066354304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765973.173535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066354304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765973.173544:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765973.173547:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765973.173550:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765973.173556:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765973.173572:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765973.173581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765973/real 1775765973] req@ffff9d81984a3100 x1862023066354432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765973.173593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066354432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765973.173612:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765973.173616:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765973.173619:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765973.173625:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765973.173642:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765973.173652:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765973/real 1775765973] req@ffff9d81984a3800 x1862023066354560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765973.173671:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066354560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765973.173686:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765973.173689:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765973.173693:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765973.173698:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765973.173714:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765973.173724:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765973/real 1775765973] req@ffff9d81984a1880 x1862023066354688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765973.173735:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066354688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775765998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765973.173743:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765973.173746:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765973.173749:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765973.813075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765974.196976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765974.837022:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765975.220980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765975.861163:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765976.244969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765976.885017:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765977.268982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765977.909065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775765978.292962:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765978.292973:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765978.292977:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775765978.292980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765978.292985:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2809 00000100:00080000:2.0:1775765978.292995:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765978.293002:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765978.293034:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066354816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765978.293057:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765978.293060:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765978.293061:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765978.293064:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2809 00000100:00080000:2.0:1775765978.293066:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765978.293068:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765978.293075:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765978.293084:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066354944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765978.293092:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765978.293095:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775765978.293095:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066354816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775765978.293099:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765978.293106:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2809 00000100:00080000:2.0:1775765978.293108:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765978.293110:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765978.293113:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765978.293123:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066355072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765978.293130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765978.293132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765978.293133:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765978.293135:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2809 00000100:00080000:2.0:1775765978.293145:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765978.293148:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765978.293151:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775765978.293155:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775765978.293161:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066355200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765978.293169:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765978.293171:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765978.293173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765978.293175:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2809 00000100:00080000:2.0:1775765978.293177:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765978.293179:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765978.293182:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765978.293191:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066355328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775765978.293198:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765978/real 1775765978] req@ffff9d81984a1500 x1862023066354816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765978.293255:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066354816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765978.293264:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765978.293267:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765978.293270:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066354816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765978.293298:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765978.293314:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765978.293322:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765978/real 1775765978] req@ffff9d81984a1180 x1862023066354944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765978.293330:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066354944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765978.293336:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765978.293338:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765978.293341:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066354944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765978.293344:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775765978.293354:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765978.293359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765978/real 1775765978] req@ffff9d81984a1f80 x1862023066355072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765978.293366:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066355072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765978.293371:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765978.293373:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765978.293375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775765978.293379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765978.293389:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765978.293395:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765978/real 1775765978] req@ffff9d81984a2300 x1862023066355200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765978.293401:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066355200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765978.293408:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765978.293409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765978.293411:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775765978.293414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775765978.293424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765978.293429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765978/real 1775765978] req@ffff9d81984a2680 x1862023066355328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765978.293436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066355328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765978.293441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775765978.293443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765978.293445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765978.933021:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765979.316968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765979.957075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765980.341009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765980.981025:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765981.364979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765982.005077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765982.388960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765983.029114:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765983.412945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765983.413009:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765983.413015:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765983.413019:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765983.413026:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2814 00000100:00080000:3.0:1775765983.413036:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765983.413042:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765983.413083:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066355456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765983.413109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765983.413112:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765983.413114:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765983.413145:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2814 00000100:00080000:3.0:1775765983.413148:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765983.413151:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765983.413160:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765983.413163:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066355456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765983.413187:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066355584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765983.413201:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765983.413205:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765983.413207:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765983.413212:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2814 00000100:00080000:3.0:1775765983.413220:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765983.413225:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765983.413242:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775765983.413247:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775765983.413259:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066355712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765983.413268:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765983.413271:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765983.413272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765983.413276:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2814 00000100:00080000:3.0:1775765983.413280:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765983.413282:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765983.413287:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775765983.413296:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765983/real 1775765983] req@ffff9d8188741880 x1862023066355456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775765983.413299:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066355840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765983.413328:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765983.413331:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765983.413333:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765983.413335:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2814 00000100:00100000:0.0:1775765983.413338:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066355456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765983.413339:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765983.413346:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765983.413352:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775765983.413356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775765983.413361:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066355968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775765983.413362:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765983.413369:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066355456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765983.413395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765983.413412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765983.413420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765983/real 1775765983] req@ffff9d8188742680 x1862023066355584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765983.413429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066355584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765983.413435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765983.413437:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765983.413439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765983.413443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765983.413453:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765983.413458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765983/real 1775765983] req@ffff9d8188743800 x1862023066355712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765983.413466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066355712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765983.413476:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765983.413478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765983.413480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765983.413483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765983.413494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765983.413500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765983/real 1775765983] req@ffff9d8188741f80 x1862023066355840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765983.413507:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066355840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765983.413513:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765983.413514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765983.413516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765983.413520:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765983.413529:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765983.413534:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765983/real 1775765983] req@ffff9d8188742300 x1862023066355968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765983.413541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066355968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765983.413546:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765983.413548:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765983.413550:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066355968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765984.053054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765984.436954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765985.077020:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765985.460977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765986.101061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765986.484947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765987.125048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765987.509016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765988.149052:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765988.532990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775765988.533046:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765988.533052:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775765988.533056:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765988.533064:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2820 00000100:00080000:2.0:1775765988.533073:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765988.533080:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775765988.533116:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066356096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765988.533188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765988.533193:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775765988.533195:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765988.533200:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2820 00000100:00080000:2.0:1775765988.533205:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765988.533211:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765988.533222:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765988.533232:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066356096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775765988.533240:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066356224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765988.533253:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765988.533256:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775765988.533258:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765988.533261:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2820 00000100:00080000:2.0:1775765988.533265:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765988.533268:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765988.533273:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775765988.533283:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066356352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765988.533291:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775765988.533295:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775765988.533297:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775765988.533300:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765988.533304:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2820 00000100:00080000:2.0:1775765988.533307:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775765988.533310:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765988.533314:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775765988.533324:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066356480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775765988.533331:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775765988.533334:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775765988.533336:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775765988.533339:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2820 00000100:00080000:2.0:1775765988.533342:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775765988.533347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765988/real 1775765988] req@ffff9d81984a0000 x1862023066356096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775765988.533347:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775765988.533350:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775765988.533364:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066356608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765988.533372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066356096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765988.533381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765988.533386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765988.533390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066356096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765988.533410:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765988.533426:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765988.533435:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765988/real 1775765988] req@ffff9d81984a2a00 x1862023066356224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765988.533444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066356224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765988.533451:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765988.533453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765988.533456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765988.533460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765988.533472:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765988.533478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765988/real 1775765988] req@ffff9d81984a2d80 x1862023066356352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765988.533486:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066356352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765988.533492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765988.533494:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765988.533496:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765988.533500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765988.533511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765988.533517:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765988/real 1775765988] req@ffff9d81984a0700 x1862023066356480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765988.533525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066356480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765988.533531:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765988.533533:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765988.533535:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765988.533539:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765988.533549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765988.533558:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765988/real 1775765988] req@ffff9d81984a3b80 x1862023066356608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765988.533572:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066356608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765988.533581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765988.533585:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765988.533589:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765989.173052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765989.557054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765990.197121:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765990.581004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765991.221004:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765991.604981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765992.245052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765992.628946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765993.269133:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765993.652981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765993.653003:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765993.653012:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765993.653016:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765993.653023:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2825 00000100:00080000:3.0:1775765993.653033:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765993.653038:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765993.653068:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066356736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765993.653107:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765993.653110:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765993.653111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765993.653114:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2825 00000100:00080000:3.0:1775765993.653117:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765993.653120:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765993.653128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765993.653140:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066356864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765993.653144:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066356736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775765993.653160:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765993.653162:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765993.653163:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765993.653166:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2825 00000100:00080000:3.0:1775765993.653168:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765993.653170:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765993.653173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765993.653180:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066356992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765993.653186:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765993.653188:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765993.653190:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765993.653192:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2825 00000100:00080000:3.0:1775765993.653204:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765993.653206:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765993.653210:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765993.653217:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066357120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765993.653224:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765993.653226:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765993.653228:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765993.653230:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2825 00000100:00080000:3.0:1775765993.653232:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765993.653235:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765993.653238:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775765993.653245:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066357248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765993.653264:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765993.653297:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765993.653320:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765993.653341:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765994.293129:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765994.676957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765995.317153:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765995.701030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765996.341139:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765996.724958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775765996.725141:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775765996.725173:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775765996.725186:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765996.725219:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765996.725228:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765996.725236:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775765996.725244:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775765996.725254:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765993/real 1775765996] req@ffff9d8188741180 x1862023066356864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765996.725278:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066356864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765996.725291:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765996.725296:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765996.725301:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765996.725307:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765993/real 1775765996] req@ffff9d8188743100 x1862023066356992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765996.725319:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066356992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765996.725326:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765996.725330:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765996.725333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066356992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765996.725338:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765993/real 1775765996] req@ffff9d8188740e00 x1862023066357120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765996.725351:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066357120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765996.725368:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765996.725371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765996.725373:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765996.725379:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765993/real 1775765996] req@ffff9d8188740a80 x1862023066357248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765996.725392:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066357248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765996.725401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765996.725404:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765996.725407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357248:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765996.725413:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765993/real 1775765996] req@ffff9d8188741500 x1862023066356736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765996.725426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066356736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765996.725434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765996.725437:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765996.725439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066356736:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775765997.365261:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765997.749002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775765998.389018:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775765998.773094:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775765998.773098:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775765998.773108:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775765998.773115:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765998.773125:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2830 00000100:00080000:3.0:1775765998.773137:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765998.773145:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775765998.773191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066357376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765998.773233:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765998.773239:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775765998.773242:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765998.773249:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2830 00000100:00080000:3.0:1775765998.773256:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765998.773263:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765998.773278:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775765998.773294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066357376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775765998.773304:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066357504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775765998.773326:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765998.773331:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775765998.773334:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765998.773342:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2830 00000100:00080000:3.0:1775765998.773348:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765998.773354:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765998.773363:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775765998.773382:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066357632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775765998.773394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775765998.773412:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765998.773418:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775765998.773422:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765998.773429:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2830 00000100:00080000:3.0:1775765998.773435:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775765998.773440:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765998.773449:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775765998.773470:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066357760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765998.773481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765998/real 1775765998] req@ffff9d8188743480 x1862023066357376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765998.773491:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775765998.773505:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775765998.773511:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775765998.773522:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2830 00000100:00080000:3.0:1775765998.773529:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775765998.773533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066357376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775765998.773540:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775765998.773554:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775765998.773565:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765998.773577:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775765998.773579:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066357888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775765998.773592:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066357376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775765998.773650:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765998.773690:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765998.773711:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765998/real 1775765998] req@ffff9d8188742d80 x1862023066357504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765998.773733:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066357504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765998.773745:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765998.773750:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765998.773756:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765998.773767:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775765998.773799:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765998.773816:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765998/real 1775765998] req@ffff9d8188743b80 x1862023066357632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765998.773845:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066357632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765998.773858:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765998.773863:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765998.773868:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775765998.773913:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765998.773946:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765998.773964:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765998/real 1775765998] req@ffff9d8188741500 x1862023066357760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765998.773981:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066357760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765998.773995:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765998.774000:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765998.774005:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775765998.774014:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775765998.774066:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775765998.774085:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775765998/real 1775765998] req@ffff9d8188740a80 x1862023066357888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775765998.774105:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066357888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775765998.774118:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775765998.774123:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775765998.774128:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066357888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775765999.412993:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775765999.797061:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766000.437121:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766000.821032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766001.461081:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766001.844945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766002.485044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766002.868981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766003.509064:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766003.892962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766003.893037:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766003.893045:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766003.893049:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766003.893059:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2835 00000100:00080000:2.0:1775766003.893070:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766003.893077:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766003.893125:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066358016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766003.893150:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766003.893152:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766003.893154:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766003.893156:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2835 00000100:00080000:2.0:1775766003.893159:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766003.893164:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766003.893172:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766003.893178:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066358016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766003.893184:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066358144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766003.893202:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766003.893204:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766003.893205:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766003.893208:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2835 00000100:00080000:2.0:1775766003.893210:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766003.893212:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766003.893215:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766003.893222:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066358272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766003.893233:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766003.893236:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766003.893237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766003.893239:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2835 00000100:00080000:2.0:1775766003.893242:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766003.893245:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766003.893246:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766003.893252:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766003.893268:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066358400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766003.893275:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766003.893278:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766003.893279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766003.893282:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2835 00000100:00080000:2.0:1775766003.893284:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766003.893286:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766003.893290:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766003.893301:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066358528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766003.893302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766003/real 1775766003] req@ffff9d81984a3480 x1862023066358016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766003.893336:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066358016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766003.893344:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766003.893347:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766003.893350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066358016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766003.893367:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766003.893382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766003.893389:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766003/real 1775766003] req@ffff9d81984a3b80 x1862023066358144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766003.893397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066358144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766003.893403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766003.893405:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766003.893417:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766003.893420:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766003.893430:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766003.893436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766003/real 1775766003] req@ffff9d81984a0700 x1862023066358272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766003.893443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066358272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766003.893452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766003.893454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766003.893456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766003.893459:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766003.893470:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766003.893484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766003/real 1775766003] req@ffff9d81984a2d80 x1862023066358400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766003.893491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066358400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766003.893496:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766003.893498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766003.893500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766003.893503:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766003.893513:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766003.893518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766003/real 1775766003] req@ffff9d81984a2a00 x1862023066358528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766003.893525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066358528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766003.893530:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766003.893532:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766003.893534:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766004.533142:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766004.917131:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766005.557122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766005.940984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766006.581087:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766006.965041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766007.605074:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766007.988964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766008.629041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766009.012973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766009.013004:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766009.013010:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766009.013014:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766009.013021:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2840 00000100:00080000:1.0:1775766009.013030:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766009.013035:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766009.013067:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066358656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766009.013089:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766009.013092:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766009.013093:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766009.013096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2840 00000100:00080000:1.0:1775766009.013098:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766009.013100:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766009.013108:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766009.013116:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066358784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766009.013122:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066358656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766009.013129:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766009.013131:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766009.013132:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766009.013136:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2840 00000100:00080000:1.0:1775766009.013138:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766009.013140:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766009.013144:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766009.013152:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066358912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766009.013157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766009.013160:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766009.013161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766009.013163:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2840 00000100:00080000:1.0:1775766009.013173:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766009.013175:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766009.013180:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775766009.013185:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766009.013196:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066359040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766009.013214:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766009.013216:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766009.013218:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766009.013220:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2840 00000100:00080000:1.0:1775766009.013223:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766009.013225:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766009.013228:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766009.013232:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766009/real 1775766009] req@ffff9d8190551880 x1862023066358656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766009.013238:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066359168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766009.013265:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066358656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766009.013288:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766009.013293:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766009.013298:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066358656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766009.013320:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766009.013344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766009.013357:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766009/real 1775766009] req@ffff9d8190550e00 x1862023066358784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766009.013371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066358784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766009.013380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766009.013384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766009.013387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766009.013392:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766009.013408:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766009.013416:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766009/real 1775766009] req@ffff9d8190553100 x1862023066358912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766009.013424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066358912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766009.013429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766009.013431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766009.013443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066358912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766009.013447:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766009.013458:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766009.013464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766009/real 1775766009] req@ffff9d8190553480 x1862023066359040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766009.013471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066359040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766009.013476:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766009.013478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766009.013480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766009.013483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766009.013493:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766009.013499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766009/real 1775766009] req@ffff9d8190551500 x1862023066359168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766009.013506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066359168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766009.013511:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766009.013513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766009.013515:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766009.653176:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766010.037083:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766010.677047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766011.061070:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766011.701097:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766012.085008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766012.725072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766013.109045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766013.749052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766014.132984:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766014.132992:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775766014.132994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766014.132996:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766014.133008:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2845 00000100:00080000:1.0:1775766014.133018:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766014.133024:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766014.133079:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066359296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766014.133115:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766014.133118:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766014.133119:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766014.133122:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2845 00000100:00080000:1.0:1775766014.133126:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766014.133130:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766014.133138:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766014.133145:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066359296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766014.133151:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066359424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766014.133159:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766014.133162:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766014.133164:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766014.133166:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2845 00000100:00080000:1.0:1775766014.133169:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766014.133171:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766014.133175:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766014.133182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066359552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766014.133188:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766014.133191:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766014.133192:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766014.133194:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2845 00000800:00000100:0.0:1775766014.133198:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766014.133198:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766014.133204:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766014.133208:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766014.133217:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066359680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766014.133223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766014.133225:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766014.133226:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766014.133230:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2845 00000100:00080000:1.0:1775766014.133233:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766014.133235:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766014.133239:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766014.133242:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766014/real 1775766014] req@ffff9d8190551500 x1862023066359296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766014.133249:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066359808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766014.133264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066359296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766014.133273:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766014.133276:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766014.133280:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066359296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766014.133296:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766014.133311:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766014.133318:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766014/real 1775766014] req@ffff9d8190553480 x1862023066359424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766014.133326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066359424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766014.133332:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766014.133334:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766014.133337:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766014.133340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766014.133349:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766014.133355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766014/real 1775766014] req@ffff9d8190553100 x1862023066359552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766014.133362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066359552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766014.133367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766014.133368:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766014.133370:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766014.133374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766014.133384:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766014.133389:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766014/real 1775766014] req@ffff9d8190550e00 x1862023066359680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766014.133396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066359680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766014.133402:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766014.133405:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766014.133407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766014.133410:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766014.133419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766014.133423:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766014/real 1775766014] req@ffff9d8190551880 x1862023066359808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766014.133430:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066359808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766014.133436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766014.133438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766014.133439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066359808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766014.773080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766015.156979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766015.796998:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766016.180977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766016.821088:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766017.204969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766017.845008:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766018.228959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766018.869017:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766019.252980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766019.252983:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766019.252990:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766019.252994:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766019.253001:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2850 00000100:00080000:2.0:1775766019.253011:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766019.253017:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766019.253050:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066359936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766019.253072:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766019.253075:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766019.253076:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766019.253079:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2850 00000100:00080000:2.0:1775766019.253081:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766019.253083:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766019.253090:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766019.253099:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066360064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766019.253107:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766019.253110:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775766019.253112:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066359936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766019.253113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766019.253121:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2850 00000100:00080000:2.0:1775766019.253124:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766019.253126:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766019.253129:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766019.253138:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066360192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766019.253145:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766019.253148:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766019.253149:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766019.253152:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2850 00000100:00080000:2.0:1775766019.253163:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766019.253166:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766019.253170:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766019.253174:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766019.253179:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066360320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766019.253187:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766019.253190:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766019.253191:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766019.253193:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2850 00000100:00080000:2.0:1775766019.253196:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766019.253198:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766019.253201:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766019.253210:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066360448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766019.253230:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766019/real 1775766019] req@ffff9d81984a0000 x1862023066359936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766019.253245:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066359936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766019.253254:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766019.253257:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766019.253260:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066359936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766019.253277:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766019.253292:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766019.253299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766019/real 1775766019] req@ffff9d81984a2680 x1862023066360064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766019.253308:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066360064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766019.253313:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766019.253316:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766019.253318:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766019.253321:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766019.253331:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766019.253336:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766019/real 1775766019] req@ffff9d81984a2300 x1862023066360192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766019.253343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066360192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766019.253348:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766019.253350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766019.253352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766019.253356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766019.253366:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766019.253371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766019/real 1775766019] req@ffff9d81984a1f80 x1862023066360320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766019.253378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066360320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766019.253384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766019.253385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766019.253387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766019.253390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766019.253399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766019.253404:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766019/real 1775766019] req@ffff9d81984a1180 x1862023066360448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766019.253411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066360448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766019.253416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766019.253418:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766019.253420:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766019.893040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766020.277086:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766020.917019:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766021.300999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766021.941050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766022.324950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766022.965170:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766023.348986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766023.988996:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766024.372983:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766024.372988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766024.372995:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766024.373001:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766024.373011:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2855 00000100:00080000:2.0:1775766024.373041:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766024.373049:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766024.373115:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066360576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766024.373147:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766024.373150:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766024.373152:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766024.373172:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2855 00000100:00080000:2.0:1775766024.373175:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766024.373179:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766024.373190:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766024.373211:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066360576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766024.373215:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066360704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766024.373227:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766024.373230:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766024.373231:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766024.373235:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2855 00000100:00080000:2.0:1775766024.373238:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766024.373240:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766024.373244:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766024.373253:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066360832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766024.373261:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766024.373264:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766024.373265:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766024.373267:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2855 00000100:00080000:2.0:1775766024.373297:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766024.373300:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766024.373305:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766024.373310:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766024.373316:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066360960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766024.373324:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766024.373327:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766024.373328:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766024.373331:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2855 00000100:00080000:2.0:1775766024.373333:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766024.373336:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766024.373339:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766024.373346:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066361088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766024.373369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766024/real 1775766024] req@ffff9d81984a1500 x1862023066360576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766024.373387:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066360576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766024.373396:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766024.373400:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766024.373403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066360576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766024.373425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766024.373441:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766024.373449:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766024/real 1775766024] req@ffff9d81984a1880 x1862023066360704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766024.373457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066360704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766024.373463:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766024.373466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766024.373468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766024.373471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766024.373484:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766024.373490:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766024/real 1775766024] req@ffff9d81984a3800 x1862023066360832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766024.373497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066360832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766024.373503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766024.373504:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766024.373506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766024.373509:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766024.373521:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766024.373526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766024/real 1775766024] req@ffff9d81984a3100 x1862023066360960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766024.373534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066360960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766024.373540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766024.373542:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766024.373543:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066360960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766024.373546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766024.373556:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766024.373561:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766024/real 1775766024] req@ffff9d81984a0e00 x1862023066361088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766024.373573:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066361088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766024.373579:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766024.373580:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766024.373582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766025.013021:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766025.396995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766026.037062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766026.420963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766027.061041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766027.445000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766028.085025:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766028.469000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766029.109119:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766029.493002:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766029.493005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766029.493016:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766029.493021:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766029.493035:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2861 00000100:00080000:2.0:1775766029.493049:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766029.493060:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766029.493109:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066361216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766029.493145:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766029.493155:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766029.493159:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766029.493165:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2861 00000100:00080000:2.0:1775766029.493172:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766029.493178:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766029.493192:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766029.493215:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066361216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766029.493215:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066361344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766029.493233:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766029.493237:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766029.493240:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766029.493245:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2861 00000100:00080000:2.0:1775766029.493251:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766029.493255:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766029.493262:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766029.493287:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066361472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766029.493301:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766029.493302:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766029.493307:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766029.493309:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766029.493315:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2861 00000100:00080000:2.0:1775766029.493320:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766029.493325:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766029.493332:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766029.493346:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066361600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766029.493362:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766029.493366:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766029.493369:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766029.493373:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766029/real 1775766029] req@ffff9d81984a0a80 x1862023066361216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766029.493375:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2861 00000100:00080000:2.0:1775766029.493380:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766029.493396:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766029.493403:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766029.493407:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066361216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766029.493417:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066361728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766029.493424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766029.493431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766029.493438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066361216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766029.493473:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766029.493508:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766029.493528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766029/real 1775766029] req@ffff9d81984a0e00 x1862023066361344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766029.493551:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066361344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766029.493565:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766029.493571:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766029.493577:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766029.493587:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766029.493621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766029.493638:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766029/real 1775766029] req@ffff9d81984a3100 x1862023066361472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766029.493651:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066361472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766029.493660:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766029.493664:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766029.493667:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766029.493673:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766029.493693:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766029.493707:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766029/real 1775766029] req@ffff9d81984a3800 x1862023066361600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766029.493719:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066361600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766029.493778:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766029.493794:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766029.493798:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766029.493805:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766029.493827:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766029.493839:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766029/real 1775766029] req@ffff9d81984a1880 x1862023066361728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766029.493852:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066361728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766029.493860:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766029.493864:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766029.493867:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766030.133029:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766030.516986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766031.157116:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766031.540946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766032.181088:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766032.564988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766033.205044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766033.588993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766034.229078:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766034.612959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766034.612981:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766034.612987:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766034.612991:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766034.612999:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2866 00000100:00080000:3.0:1775766034.613007:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766034.613013:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766034.613046:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066361856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766034.613069:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766034.613072:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766034.613073:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766034.613076:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2866 00000100:00080000:3.0:1775766034.613078:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766034.613081:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766034.613089:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766034.613098:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066361984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766034.613102:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066361856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766034.613114:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766034.613116:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766034.613117:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766034.613120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2866 00000100:00080000:3.0:1775766034.613124:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766034.613126:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766034.613129:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766034.613145:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066362112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766034.613152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766034.613155:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766034.613156:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766034.613158:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2866 00000100:00080000:3.0:1775766034.613161:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766034.613165:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766034.613167:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766034.613171:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766034.613181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066362240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766034.613188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766034.613190:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766034.613192:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766034.613195:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2866 00000100:00080000:3.0:1775766034.613197:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766034.613200:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766034.613203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766034.613209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066362368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766034.613220:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766034/real 1775766034] req@ffff9d8188740e00 x1862023066361856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766034.613237:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066361856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766034.613246:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766034.613249:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766034.613252:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066361856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766034.613269:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766034.613289:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766034.613302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766034/real 1775766034] req@ffff9d8188743100 x1862023066361984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766034.613314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066361984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766034.613322:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766034.613326:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766034.613329:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066361984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766034.613335:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766034.613355:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766034.613364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766034/real 1775766034] req@ffff9d8188741180 x1862023066362112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766034.613373:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066362112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766034.613378:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766034.613380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766034.613383:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766034.613387:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766034.613399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766034.613405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766034/real 1775766034] req@ffff9d8188742300 x1862023066362240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766034.613412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066362240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766034.613418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766034.613420:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766034.613422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766034.613425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766034.613435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766034.613440:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766034/real 1775766034] req@ffff9d8188741f80 x1862023066362368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766034.613447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066362368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766034.613452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766034.613454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766034.613456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766035.253052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766035.636966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766036.277115:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766036.660959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766037.301151:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766037.684972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766038.325138:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766038.709017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766039.349213:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766039.733110:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766039.733113:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766039.733125:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766039.733131:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766039.733143:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2871 00000100:00080000:2.0:1775766039.733154:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766039.733162:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766039.733205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066362496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766039.733238:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766039.733242:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766039.733244:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766039.733248:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2871 00000100:00080000:2.0:1775766039.733262:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766039.733266:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766039.733276:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766039.733288:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066362624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766039.733299:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766039.733303:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766039.733305:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766039.733309:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2871 00000100:00080000:2.0:1775766039.733312:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766039.733319:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066362496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766039.733320:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766039.733329:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766039.733340:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066362752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766039.733351:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766039.733355:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766039.733357:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766039.733360:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2871 00000100:00080000:2.0:1775766039.733363:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766039.733367:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766039.733372:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766039.733382:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066362880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766039.733398:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766039.733402:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766039.733404:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766039.733412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766039.733413:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2871 00000100:00080000:2.0:1775766039.733422:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766039.733425:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766039.733431:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766039.733443:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066363008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766039.733501:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766039/real 1775766039] req@ffff9d81984a1500 x1862023066362496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766039.733545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066362496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766039.733563:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766039.733570:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766039.733578:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066362496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766039.733590:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766039.733627:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766039.733705:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766039/real 1775766039] req@ffff9d81984a1180 x1862023066362624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766039.733729:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066362624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766039.733743:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766039.733749:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766039.733755:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766039.733790:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766039.733829:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766039.733847:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766039/real 1775766039] req@ffff9d81984a1f80 x1862023066362752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766039.733868:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066362752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766039.733930:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766039.733937:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766039.733943:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766039.733964:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766039.733998:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766039.734017:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766039/real 1775766039] req@ffff9d81984a2300 x1862023066362880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766039.734038:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066362880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766039.734054:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766039.734071:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766039.734076:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066362880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766039.734085:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766039.734117:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766039.734134:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766039/real 1775766039] req@ffff9d81984a2680 x1862023066363008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766039.734164:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066363008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766039.734177:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766039.734182:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766039.734188:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766040.373066:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766040.756961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766041.397061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766041.780999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766042.421008:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766042.804971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766043.445052:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766043.829036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766044.469070:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766044.852931:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766044.852939:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766044.852943:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766044.852945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766044.852951:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2876 00000100:00080000:2.0:1775766044.852960:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766044.852966:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766044.852999:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066363136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766044.853022:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766044.853025:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766044.853026:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766044.853029:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2876 00000100:00080000:2.0:1775766044.853031:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766044.853034:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766044.853042:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766044.853053:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066363264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766044.853063:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066363136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766044.853064:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766044.853068:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766044.853071:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766044.853076:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2876 00000100:00080000:2.0:1775766044.853078:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766044.853080:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766044.853083:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766044.853092:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066363392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766044.853099:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766044.853102:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766044.853103:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766044.853106:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2876 00000100:00080000:2.0:1775766044.853119:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766044.853122:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766044.853125:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766044.853131:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766044.853134:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066363520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766044.853142:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766044.853144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766044.853146:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766044.853150:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2876 00000100:00080000:2.0:1775766044.853153:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766044.853157:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766044.853161:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766044.853172:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066363648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766044.853186:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766044/real 1775766044] req@ffff9d81984a0000 x1862023066363136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766044.853203:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066363136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766044.853211:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766044.853215:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766044.853219:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066363136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766044.853237:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766044.853254:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766044.853261:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766044/real 1775766044] req@ffff9d81984a2a00 x1862023066363264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766044.853278:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066363264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766044.853284:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766044.853286:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766044.853288:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766044.853292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766044.853304:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766044.853310:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766044/real 1775766044] req@ffff9d81984a2d80 x1862023066363392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766044.853317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066363392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766044.853322:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766044.853324:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766044.853326:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766044.853329:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766044.853348:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766044.853355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766044/real 1775766044] req@ffff9d81984a0700 x1862023066363520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766044.853362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066363520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766044.853373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766044.853375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766044.853381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766044.853384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766044.853395:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766044.853400:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766044/real 1775766044] req@ffff9d81984a3b80 x1862023066363648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766044.853407:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066363648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766044.853413:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766044.853414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766044.853416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766045.493035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766045.877034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766046.517072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766046.901029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766047.541039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766047.924994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766048.565122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766048.948964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766049.589153:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766049.972999:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766049.973007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766049.973011:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766049.973018:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766049.973029:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2881 00000100:00080000:2.0:1775766049.973042:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766049.973049:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766049.973095:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066363776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766049.973124:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766049.973127:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766049.973129:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766049.973132:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2881 00000100:00080000:2.0:1775766049.973135:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766049.973139:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766049.973150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766049.973160:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066363776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766049.973164:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066363904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766049.973183:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766049.973187:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766049.973189:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766049.973194:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2881 00000100:00080000:2.0:1775766049.973196:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766049.973199:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766049.973204:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766049.973214:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066364032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766049.973220:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766049.973223:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766049.973224:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766049.973227:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2881 00000100:00080000:2.0:1775766049.973230:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766049.973234:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766049.973239:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766049.973251:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766049.973262:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066364160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766049.973269:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766049.973272:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766049.973273:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766049.973279:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2881 00000100:00080000:2.0:1775766049.973282:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766049.973289:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766049.973293:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766049.973299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766049/real 1775766049] req@ffff9d81984a3480 x1862023066363776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766049.973303:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066364288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766049.973383:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066363776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766049.973391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766049.973395:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766049.973398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066363776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766049.973417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766049.973436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766049.973445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766049/real 1775766049] req@ffff9d81984a3b80 x1862023066363904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766049.973454:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066363904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766049.973460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766049.973462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766049.973464:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066363904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766049.973468:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766049.973479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766049.973484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766049/real 1775766049] req@ffff9d81984a0700 x1862023066364032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766049.973492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066364032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766049.973501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766049.973504:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766049.973506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766049.973509:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766049.973522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766049.973532:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766049/real 1775766049] req@ffff9d81984a2d80 x1862023066364160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766049.973581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066364160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766049.973591:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766049.973595:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766049.973598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766049.973605:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766049.973626:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766049.973637:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766049/real 1775766049] req@ffff9d81984a2a00 x1862023066364288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766049.973650:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066364288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766049.973659:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766049.973663:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766049.973666:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766050.613076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766050.996980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766051.637008:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766052.020959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766052.661072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766053.045006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766053.685125:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766054.069003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766054.709185:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766055.093051:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766055.093062:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766055.093072:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766055.093079:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766055.093092:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2886 00000100:00080000:1.0:1775766055.093105:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766055.093115:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766055.093164:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066364416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766055.093220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766055.093228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766055.093230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766055.093235:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2886 00000100:00080000:1.0:1775766055.093240:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766055.093245:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766055.093258:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766055.093279:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066364544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766055.093281:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066364416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766055.093306:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766055.093310:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766055.093312:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766055.093319:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2886 00000100:00080000:1.0:1775766055.093323:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766055.093327:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766055.093333:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766055.093347:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066364672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766055.093358:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766055.093361:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766055.093363:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766055.093368:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2886 00000800:00000100:0.0:1775766055.093375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766055.093379:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766055.093386:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766055.093394:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766055.093408:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066364800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766055.093419:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766055.093423:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766055.093426:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766055.093432:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2886 00000100:00080000:1.0:1775766055.093437:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766055.093445:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766055.093446:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766055/real 1775766055] req@ffff9d8190551880 x1862023066364416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766055.093453:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766055.093476:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066364928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766055.093486:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066364416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766055.093578:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766055.093585:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766055.093592:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066364416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766055.093621:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766055.093662:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766055.093678:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766055/real 1775766055] req@ffff9d8190550e00 x1862023066364544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766055.093693:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066364544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766055.093703:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766055.093707:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766055.093712:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766055.093718:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766055.093738:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766055.093747:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766055/real 1775766055] req@ffff9d8190553100 x1862023066364672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766055.093760:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066364672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766055.093769:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766055.093772:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766055.093775:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766055.093782:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766055.093800:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766055.093809:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766055/real 1775766055] req@ffff9d8190553480 x1862023066364800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766055.093822:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066364800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766055.093832:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766055.093835:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766055.093838:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766055.093844:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766055.093860:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766055.093869:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766055/real 1775766055] req@ffff9d8190551500 x1862023066364928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766055.093915:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066364928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766055.093925:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766055.093946:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766055.093951:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066364928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766055.733182:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766056.117013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766056.757130:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766057.141009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766057.781144:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766058.164982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766058.805138:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766059.188980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766059.829060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766060.212967:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766060.212968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766060.212976:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766060.212980:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766060.212989:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2891 00000100:00080000:1.0:1775766060.212998:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766060.213003:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766060.213039:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066365056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766060.213062:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766060.213065:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766060.213066:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766060.213068:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2891 00000100:00080000:1.0:1775766060.213071:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766060.213074:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766060.213081:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766060.213090:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066365184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766060.213098:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766060.213101:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066365056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766060.213101:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766060.213104:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766060.213111:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2891 00000100:00080000:1.0:1775766060.213113:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766060.213116:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766060.213119:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766060.213127:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066365312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766060.213134:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766060.213136:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766060.213137:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766060.213140:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2891 00000100:00080000:1.0:1775766060.213152:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766060.213154:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766060.213159:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766060.213165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066365440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766060.213176:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766060.213178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766060.213180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766060.213183:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2891 00000100:00080000:1.0:1775766060.213185:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766060.213188:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766060.213192:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766060.213200:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066365568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766060.213204:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766060.213237:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766060.213250:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766060.213267:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766060.852996:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766061.236971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766061.877015:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766062.260952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766062.901181:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766063.285012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775766063.285211:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775766063.285254:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775766063.285275:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766063.285364:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766063.285382:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766063.285392:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766063.285403:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766063.285520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766060/real 1775766063] req@ffff9d8190551880 x1862023066365568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766063.285568:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066365568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766063.285586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766063.285594:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766063.285602:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365568:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766063.285613:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766060/real 1775766063] req@ffff9d8190553480 x1862023066365184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766063.285634:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066365184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766063.285646:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766063.285651:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766063.285656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766063.285666:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766060/real 1775766063] req@ffff9d8190553100 x1862023066365312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766063.285708:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066365312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766063.285737:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766063.285743:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766063.285748:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766063.285759:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766060/real 1775766063] req@ffff9d8190550e00 x1862023066365440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766063.285774:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066365440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766063.285785:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766063.285790:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766063.285796:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766063.285804:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766060/real 1775766063] req@ffff9d8190551500 x1862023066365056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766063.285822:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066365056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766063.285833:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766063.285838:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766063.285843:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066365056:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775766063.925012:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766064.308976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766064.949147:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766065.333107:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766065.333123:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1775766065.333130:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766065.333131:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766065.333154:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2896 00000100:00080000:2.0:1775766065.333172:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766065.333184:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766065.333231:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066365696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766065.333289:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766065.333293:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766065.333295:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766065.333303:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2896 00000100:00080000:2.0:1775766065.333308:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766065.333313:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766065.333326:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766065.333344:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066365824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766065.333360:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766065.333365:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766065.333368:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766065.333374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066365696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766065.333383:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2896 00000100:00080000:2.0:1775766065.333432:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766065.333439:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766065.333446:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766065.333466:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066365952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766065.333480:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766065.333484:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766065.333487:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766065.333495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766065.333496:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2896 00000100:00080000:2.0:1775766065.333505:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766065.333512:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766065.333521:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766065.333540:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066366080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766065.333555:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766065.333562:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766065/real 1775766065] req@ffff9d81984a0000 x1862023066365696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766065.333563:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766065.333570:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766065.333584:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2896 00000100:00080000:2.0:1775766065.333593:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766065.333598:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766065.333605:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066365696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766065.333607:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775766065.333628:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766065.333634:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066366208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766065.333641:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766065.333653:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066365696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766065.333693:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766065.333732:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766065.333755:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766065/real 1775766065] req@ffff9d81984a2680 x1862023066365824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766065.333783:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066365824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766065.333795:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766065.333800:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766065.333804:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766065.333813:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766065.333843:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766065.333858:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766065/real 1775766065] req@ffff9d81984a2300 x1862023066365952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766065.333906:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066365952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766065.333917:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766065.333922:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766065.333926:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066365952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766065.333935:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766065.333964:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766065.333980:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766065/real 1775766065] req@ffff9d81984a1f80 x1862023066366080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766065.333998:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066366080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766065.334018:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766065.334023:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766065.334027:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766065.334035:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766065.334061:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766065.334075:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766065/real 1775766065] req@ffff9d81984a1180 x1862023066366208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766065.334090:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066366208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766065.334100:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766065.334105:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766065.334109:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766065.973142:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766066.357107:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766066.997161:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766067.381058:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766068.021090:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766068.404991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766069.045121:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766069.428980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766070.069112:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766070.453021:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766070.453035:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766070.453042:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766070.453055:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2901 00000100:00100000:3.0:1775766070.453066:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766070.453068:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766070.453082:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766070.453131:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066366336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766070.453164:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766070.453169:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766070.453171:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766070.453176:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2901 00000100:00080000:2.0:1775766070.453181:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766070.453187:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766070.453199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766070.453222:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066366336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766070.453223:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066366464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766070.453252:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766070.453258:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766070.453261:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766070.453269:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2901 00000100:00080000:2.0:1775766070.453276:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766070.453283:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766070.453292:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766070.453322:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066366592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766070.453323:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766070.453380:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766070.453385:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766070.453387:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766070.453393:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2901 00000100:00080000:2.0:1775766070.453399:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766070.453405:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766070.453413:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766070.453434:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766070/real 1775766070] req@ffff9d81984a1500 x1862023066366336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766070.453436:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066366720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766070.453458:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766070.453475:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766070.453478:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766070.453487:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066366336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766070.453488:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2901 00000100:00080000:2.0:1775766070.453499:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766070.453511:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775766070.453518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766070.453520:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775766070.453531:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:2.0:1775766070.453538:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066366848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766070.453542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066366336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766070.453584:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766070.453634:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766070.453649:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766070/real 1775766070] req@ffff9d81984a1880 x1862023066366464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766070.453665:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066366464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766070.453681:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766070.453687:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766070.453693:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766070.453703:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766070.453739:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766070.453759:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766070/real 1775766070] req@ffff9d81984a3800 x1862023066366592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766070.453782:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066366592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766070.453795:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766070.453799:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766070.453803:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766070.453810:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766070.453836:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766070.453849:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766070/real 1775766070] req@ffff9d81984a3100 x1862023066366720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766070.453862:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066366720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766070.453872:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766070.453907:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766070.453911:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766070.453918:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766070.453939:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766070.453951:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766070/real 1775766070] req@ffff9d81984a0e00 x1862023066366848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766070.453965:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066366848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766070.453974:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766070.453978:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766070.453982:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066366848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766071.093147:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766071.477005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766072.117056:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766072.500981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766073.141143:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766073.524958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766074.165082:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766074.548941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766075.189158:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766075.573058:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766075.573226:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766075.573234:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766075.573239:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766075.573249:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2907 00000100:00080000:1.0:1775766075.573262:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766075.573270:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766075.573361:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066366976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766075.573388:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766075.573392:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766075.573393:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766075.573397:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2907 00000100:00080000:1.0:1775766075.573400:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766075.573404:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766075.573425:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766075.573437:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066367104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766075.573449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066366976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766075.573452:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766075.573459:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766075.573461:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766075.573466:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2907 00000100:00080000:1.0:1775766075.573468:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766075.573471:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766075.573476:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766075.573486:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066367232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766075.573495:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766075.573499:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766075.573500:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766075.573503:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2907 00000100:00080000:1.0:1775766075.573509:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766075.573511:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766075.573515:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766075.573520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766075.573528:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066367360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766075.573539:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766075.573541:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766075.573543:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766075.573548:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2907 00000100:00080000:1.0:1775766075.573551:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766075.573556:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766075.573560:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766075.573571:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766075/real 1775766075] req@ffff9d8190551500 x1862023066366976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766075.573573:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066367488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766075.573639:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066366976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766075.573653:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766075.573660:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766075.573664:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066366976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766075.573689:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766075.573718:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766075.573734:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766075/real 1775766075] req@ffff9d8190550e00 x1862023066367104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766075.573750:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066367104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766075.573760:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766075.573763:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766075.573767:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766075.573774:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766075.573797:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766075.573810:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766075/real 1775766075] req@ffff9d8190553100 x1862023066367232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766075.573842:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066367232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766075.573864:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766075.573869:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766075.573875:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766075.573883:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766075.573969:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766075.573988:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766075/real 1775766075] req@ffff9d8190553480 x1862023066367360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766075.574008:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066367360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766075.574025:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766075.574031:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766075.574036:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766075.574047:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766075.574075:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766075.574093:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766075/real 1775766075] req@ffff9d8190551880 x1862023066367488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766075.574114:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066367488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766075.574129:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766075.574134:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766075.574141:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766076.213144:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766076.596964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766077.237096:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766077.620983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766078.261043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766078.645004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766079.285129:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766079.669017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766080.309079:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766080.693031:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766080.693032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766080.693047:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766080.693053:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766080.693066:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2912 00000100:00080000:1.0:1775766080.693080:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766080.693090:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766080.693148:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066367616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766080.693209:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766080.693217:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766080.693221:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766080.693229:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2912 00000100:00080000:1.0:1775766080.693238:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766080.693282:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766080.693300:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766080.693316:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066367616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766080.693334:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066367744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766080.693385:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766080.693392:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766080.693397:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766080.693404:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2912 00000100:00080000:1.0:1775766080.693412:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766080.693413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766080.693426:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766080.693437:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766080.693462:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066367872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766080.693477:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766080.693485:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775766080.693488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766080/real 1775766080] req@ffff9d8190551880 x1862023066367616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766080.693489:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766080.693511:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2912 00000100:00080000:1.0:1775766080.693523:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766080.693534:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766080.693542:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766080.693549:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066367616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766080.693564:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066368000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766080.693569:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766080.693586:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:1.0:1775766080.693593:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766080.693601:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066367616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766080.693618:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766080.693623:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766080.693630:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2912 00000100:00080000:1.0:1775766080.693638:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766080.693646:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367744:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775766080.693654:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766080.693671:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766080.693689:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066368128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766080.693721:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766080.693753:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766080/real 1775766080] req@ffff9d8190553480 x1862023066367744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766080.693771:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066367744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766080.693781:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766080.693786:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766080.693790:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766080.693798:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766080.693824:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766080.693836:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766080/real 1775766080] req@ffff9d8190553100 x1862023066367872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766080.693850:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066367872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766080.693859:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766080.693863:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766080.693866:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066367872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766080.693918:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766080.693944:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766080.693958:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766080/real 1775766080] req@ffff9d8190550e00 x1862023066368000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766080.693972:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066368000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766080.693984:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766080.693988:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766080.693992:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766080.694011:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766080.694031:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766080.694041:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766080/real 1775766080] req@ffff9d8190551500 x1862023066368128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766080.694064:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066368128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766080.694074:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766080.694078:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766080.694081:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766081.333152:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766081.717001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766082.357138:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766082.741133:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766083.381170:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766083.765002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766084.405158:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766084.789042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766085.429060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766085.813021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766085.813083:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766085.813116:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766085.813121:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766085.813132:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2917 00000100:00080000:2.0:1775766085.813145:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766085.813154:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766085.813235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066368256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766085.813277:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766085.813281:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766085.813283:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766085.813287:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2917 00000100:00080000:2.0:1775766085.813290:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766085.813294:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766085.813307:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766085.813323:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066368384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766085.813333:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766085.813336:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766085.813338:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766085.813342:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2917 00000100:00080000:2.0:1775766085.813345:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766085.813349:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766085.813354:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766085.813371:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066368512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766085.813396:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766085.813399:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766085.813400:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766085.813403:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2917 00000100:00080000:2.0:1775766085.813406:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766085.813410:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766085.813415:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766085.813431:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066368640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766085.813436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066368256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766085.813441:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766085.813457:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766085.813461:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766085.813466:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2917 00000100:00080000:2.0:1775766085.813468:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766085.813471:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766085.813474:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766085.813485:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066368768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766085.813549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766085.813609:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766085/real 1775766085] req@ffff9d81984a0a80 x1862023066368256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766085.813635:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066368256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766085.813649:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766085.813654:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766085.813660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066368256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766085.813670:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766085.813705:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766085.813718:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766085/real 1775766085] req@ffff9d81984a0e00 x1862023066368384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766085.813733:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066368384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766085.813752:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766085.813756:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766085.813760:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766085.813767:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766085.813786:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766085.813797:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766085/real 1775766085] req@ffff9d81984a3100 x1862023066368512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766085.813810:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066368512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766085.813820:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766085.813824:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766085.813828:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766085.813870:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766085.813958:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766085.813973:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766085/real 1775766085] req@ffff9d81984a3800 x1862023066368640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766085.813993:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066368640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766085.814012:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766085.814019:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766085.814026:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766085.814038:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766085.814075:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766085.814096:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766085/real 1775766085] req@ffff9d81984a1880 x1862023066368768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766085.814127:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066368768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766085.814139:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766085.814143:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766085.814147:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066368768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766086.453167:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766086.837055:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766087.477217:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766087.860990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766088.501122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766088.884973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766089.525051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766089.908956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766090.549019:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766090.932957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766090.933005:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766090.933013:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766090.933019:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766090.933046:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2922 00000100:00080000:1.0:1775766090.933075:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766090.933082:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766090.933134:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066368896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766090.933202:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766090.933206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766090.933208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766090.933212:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2922 00000100:00080000:1.0:1775766090.933216:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766090.933219:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766090.933229:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766090.933244:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066369024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766090.933256:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766090.933257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066368896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766090.933268:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766090.933270:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766090.933274:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2922 00000100:00080000:1.0:1775766090.933278:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766090.933281:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766090.933286:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766090.933296:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066369152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766090.933303:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766090.933306:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766090.933308:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766090.933317:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766090.933320:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2922 00000100:00080000:1.0:1775766090.933323:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766090.933327:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766090.933331:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766090.933340:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066369280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766090.933347:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766090.933349:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766090.933351:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766090.933354:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2922 00000100:00080000:1.0:1775766090.933357:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766090.933361:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766090.933362:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766090/real 1775766090] req@ffff9d8190551500 x1862023066368896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766090.933367:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766090.933378:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066369408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766090.933385:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066368896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766090.933395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766090.933398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766090.933403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066368896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766090.933419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766090.933434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766090.933441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766090/real 1775766090] req@ffff9d8190550e00 x1862023066369024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766090.933449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066369024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766090.933455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766090.933457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766090.933459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766090.933462:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766090.933472:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766090.933488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766090/real 1775766090] req@ffff9d8190553100 x1862023066369152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766090.933495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066369152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766090.933501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766090.933503:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766090.933505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766090.933508:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766090.933519:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766090.933524:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766090/real 1775766090] req@ffff9d8190553480 x1862023066369280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766090.933536:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066369280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766090.933558:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766090.933560:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766090.933562:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766090.933566:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766090.933576:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766090.933582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766090/real 1775766090] req@ffff9d8190551880 x1862023066369408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766090.933590:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066369408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766090.933595:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766090.933596:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766090.933598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766091.573162:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766091.956953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766092.597201:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766092.981012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766093.621173:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766094.004987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766094.645108:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766095.029022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766095.669032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766096.053022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766096.053048:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766096.053059:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766096.053081:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766096.053092:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2927 00000100:00080000:2.0:1775766096.053129:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766096.053138:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766096.053178:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066369536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766096.053205:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766096.053208:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766096.053210:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766096.053213:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2927 00000100:00080000:2.0:1775766096.053216:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766096.053221:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766096.053232:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766096.053248:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066369664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766096.053260:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066369536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766096.053282:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766096.053286:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766096.053289:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766096.053295:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2927 00000100:00080000:2.0:1775766096.053301:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766096.053304:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766096.053310:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766096.053322:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066369792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766096.053331:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766096.053337:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766096.053340:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766096.053342:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766096.053346:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2927 00000100:00080000:2.0:1775766096.053352:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766096.053355:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766096.053359:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766096.053372:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066369920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766096.053380:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766096.053383:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766096/real 1775766096] req@ffff9d81984a1500 x1862023066369536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766096.053384:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766096.053388:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766096.053396:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2927 00000100:00080000:2.0:1775766096.053403:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766096.053407:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766096.053412:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766096.053416:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066369536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766096.053424:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066370048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766096.053432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766096.053440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766096.053445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066369536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766096.053466:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766096.053485:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766096.053494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766096/real 1775766096] req@ffff9d81984a1180 x1862023066369664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766096.053504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066369664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766096.053511:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766096.053514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766096.053516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766096.053520:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766096.053532:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766096.053538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766096/real 1775766096] req@ffff9d81984a1f80 x1862023066369792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766096.053547:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066369792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766096.053553:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766096.053555:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766096.053558:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766096.053563:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766096.053575:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766096.053582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766096/real 1775766096] req@ffff9d81984a2300 x1862023066369920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766096.053590:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066369920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766096.053597:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766096.053599:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766096.053602:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066369920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766096.053606:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766096.053619:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766096.053625:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766096/real 1775766096] req@ffff9d81984a2680 x1862023066370048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766096.053634:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066370048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766096.053640:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766096.053642:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766096.053645:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766096.693220:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766097.077001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766097.717191:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766098.100953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766098.741187:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766099.124993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766099.765151:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766100.149009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766100.789254:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766101.172985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766101.173013:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766101.173023:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766101.173030:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766101.173043:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2932 00000100:00080000:2.0:1775766101.173086:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766101.173095:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766101.173143:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066370176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766101.173180:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766101.173185:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766101.173187:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766101.173192:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2932 00000100:00080000:2.0:1775766101.173196:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766101.173213:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766101.173225:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766101.173243:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066370304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766101.173259:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766101.173263:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066370176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766101.173269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766101.173272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766101.173278:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2932 00000100:00080000:2.0:1775766101.173282:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766101.173286:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766101.173292:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766101.173302:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066370432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766101.173325:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766101.173330:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766101.173332:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766101.173338:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2932 00000100:00080000:2.0:1775766101.173341:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766101.173345:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766101.173351:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766101.173365:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066370560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766101.173377:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766101.173381:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766101.173383:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766101.173388:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2932 00000100:00080000:2.0:1775766101.173392:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766101.173396:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766101.173404:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766101.173411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766101.173425:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066370688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766101.173487:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766101/real 1775766101] req@ffff9d81984a0000 x1862023066370176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766101.173513:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066370176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766101.173527:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766101.173534:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766101.173541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066370176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766101.173567:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766101.173595:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766101.173609:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766101/real 1775766101] req@ffff9d81984a2a00 x1862023066370304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766101.173625:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066370304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766101.173635:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766101.173639:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766101.173643:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766101.173650:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766101.173668:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766101.173678:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766101/real 1775766101] req@ffff9d81984a2d80 x1862023066370432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766101.173700:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066370432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766101.173710:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766101.173713:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766101.173717:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766101.173723:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766101.173741:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766101.173751:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766101/real 1775766101] req@ffff9d81984a0700 x1862023066370560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766101.173765:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066370560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766101.173775:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766101.173779:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766101.173782:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766101.173788:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766101.173805:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766101.173814:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766101/real 1775766101] req@ffff9d81984a3b80 x1862023066370688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766101.173827:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066370688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766101.173845:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766101.173849:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766101.173853:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766101.813112:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766102.197022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766102.837131:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766103.221001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766103.861061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766104.245084:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766104.885121:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766105.269071:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766105.909172:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766106.293083:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766106.293085:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766106.293099:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766106.293105:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766106.293115:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2937 00000100:00080000:3.0:1775766106.293126:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766106.293132:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766106.293173:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066370816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766106.293201:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766106.293205:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766106.293207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766106.293210:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2937 00000100:00080000:3.0:1775766106.293214:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766106.293218:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766106.293227:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766106.293240:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066370944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766106.293247:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066370816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766106.293259:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766106.293264:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766106.293266:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766106.293271:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2937 00000100:00080000:3.0:1775766106.293274:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766106.293277:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766106.293281:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766106.293294:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066371072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766106.293302:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766106.293305:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766106.293307:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766106.293310:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2937 00000100:00080000:3.0:1775766106.293313:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766106.293321:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766106.293322:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766106.293328:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766106.293341:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066371200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766106.293350:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766106.293353:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766106.293355:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766106.293359:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2937 00000100:00080000:3.0:1775766106.293362:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766106.293366:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766106.293371:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766106.293377:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766106/real 1775766106] req@ffff9d8188743800 x1862023066370816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766106.293385:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066371328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766106.293410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066370816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766106.293422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766106.293427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766106.293431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066370816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766106.293467:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766106.293501:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766106.293519:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766106/real 1775766106] req@ffff9d8188742680 x1862023066370944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766106.293538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066370944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766106.293551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766106.293556:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766106.293559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066370944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766106.293565:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766106.293586:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766106.293596:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766106/real 1775766106] req@ffff9d8188741880 x1862023066371072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766106.293607:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066371072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766106.293614:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766106.293617:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766106.293620:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766106.293624:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766106.293638:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766106.293645:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766106/real 1775766106] req@ffff9d8188741c00 x1862023066371200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766106.293655:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066371200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766106.293663:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766106.293665:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766106.293668:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766106.293672:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766106.293697:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766106.293710:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766106/real 1775766106] req@ffff9d8188742a00 x1862023066371328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766106.293720:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066371328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766106.293728:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766106.293731:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766106.293733:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766106.933110:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766107.317156:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766107.957067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766108.341057:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766108.981077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766109.364925:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766110.005180:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766110.389014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766111.029138:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766111.413002:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766111.413011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766111.413012:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766111.413020:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766111.413030:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2942 00000100:00080000:1.0:1775766111.413042:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766111.413049:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766111.413088:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066371456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766111.413115:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766111.413119:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766111.413121:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766111.413124:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2942 00000100:00080000:1.0:1775766111.413127:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766111.413135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766111.413144:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766111.413156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066371584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766111.413166:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766111.413174:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766111.413176:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766111.413180:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2942 00000100:00080000:1.0:1775766111.413185:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766111.413191:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066371456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766111.413192:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766111.413203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766111.413224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066371712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766111.413241:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766111.413246:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766111.413248:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766111.413254:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2942 00000100:00080000:1.0:1775766111.413260:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766111.413266:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766111.413271:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766111.413280:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766111.413296:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066371840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766111.413306:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766111.413310:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766111.413312:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766111.413318:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766111/real 1775766111] req@ffff9d8190551880 x1862023066371456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766111.413318:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2942 00000100:00080000:1.0:1775766111.413326:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766111.413335:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766111.413342:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766111.413351:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066371456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766111.413357:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066371968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766111.413377:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766111.413388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766111.413395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066371456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766111.413404:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766111.413426:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766111.413437:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766111/real 1775766111] req@ffff9d8190553480 x1862023066371584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766111.413459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066371584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766111.413467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766111.413470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766111.413473:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766111.413493:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766111.413508:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766111.413523:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766111/real 1775766111] req@ffff9d8190553100 x1862023066371712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766111.413534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066371712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766111.413541:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766111.413544:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766111.413547:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766111.413551:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766111.413564:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766111.413571:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766111/real 1775766111] req@ffff9d8190550e00 x1862023066371840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766111.413581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066371840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766111.413595:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766111.413598:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766111.413600:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766111.413605:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766111.413625:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766111.413632:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766111/real 1775766111] req@ffff9d8190551500 x1862023066371968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766111.413642:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066371968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766111.413649:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766111.413652:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766111.413654:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066371968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766112.053043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766112.436978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766113.077044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766113.460973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766114.101147:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766114.485028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766115.125067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766115.509032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766116.149125:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766116.532973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766116.533038:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766116.533046:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766116.533051:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766116.533062:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2948 00000100:00080000:1.0:1775766116.533077:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766116.533088:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766116.533144:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066372096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766116.533171:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766116.533174:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766116.533176:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766116.533179:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2948 00000100:00080000:1.0:1775766116.533183:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766116.533186:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766116.533196:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766116.533208:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066372224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766116.533220:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766116.533223:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766116.533225:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766116.533234:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066372096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766116.533239:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2948 00000100:00080000:1.0:1775766116.533248:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766116.533251:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766116.533256:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766116.533267:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066372352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766116.533275:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766116.533278:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766116.533279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766116.533283:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2948 00000100:00080000:1.0:1775766116.533297:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766116.533300:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766116.533304:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766116.533311:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766116.533315:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066372480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766116.533324:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766116.533327:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766116.533329:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766116.533336:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2948 00000100:00080000:1.0:1775766116.533339:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766116.533341:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766116.533345:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766116.533357:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066372608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766116.533367:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766116/real 1775766116] req@ffff9d8190551500 x1862023066372096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766116.533385:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066372096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766116.533395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766116.533399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766116.533402:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066372096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766116.533421:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766116.533438:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766116.533445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766116/real 1775766116] req@ffff9d8190550e00 x1862023066372224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766116.533454:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066372224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766116.533459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766116.533462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766116.533464:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766116.533469:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766116.533487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766116.533498:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766116/real 1775766116] req@ffff9d8190553100 x1862023066372352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766116.533511:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066372352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766116.533520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766116.533524:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766116.533527:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766116.533534:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766116.533551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766116.533559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766116/real 1775766116] req@ffff9d8190553480 x1862023066372480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766116.533567:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066372480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766116.533574:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766116.533576:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766116.533578:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766116.533581:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766116.533591:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766116.533596:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766116/real 1775766116] req@ffff9d8190551880 x1862023066372608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766116.533604:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066372608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766116.533609:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766116.533611:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766116.533613:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766117.173075:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766117.557085:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766118.197159:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766118.581020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766119.221057:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766119.605025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766120.245053:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766120.628970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766121.269106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766121.653091:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766121.653141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766121.653153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766121.653161:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766121.653177:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2953 00000100:00080000:1.0:1775766121.653195:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766121.653208:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766121.653279:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066372736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766121.653328:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766121.653335:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766121.653339:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766121.653347:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2953 00000100:00080000:1.0:1775766121.653354:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766121.653361:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766121.653380:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766121.653408:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066372864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766121.653430:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766121.653436:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766121.653440:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766121.653447:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2953 00000100:00080000:1.0:1775766121.653453:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766121.653464:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766121.653474:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766121.653484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066372736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766121.653495:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066372992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766121.653529:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766121.653535:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766121.653538:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766121.653544:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2953 00000100:00080000:1.0:1775766121.653550:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766121.653557:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766121.653564:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766121.653579:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066373120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766121.653591:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766121.653606:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1775766121.653607:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766121.653617:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766121.653626:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2953 00000100:00080000:1.0:1775766121.653635:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766121.653640:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766121.653650:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766121.653666:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066373248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766121.653691:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766121/real 1775766121] req@ffff9d8190551880 x1862023066372736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766121.653727:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066372736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766121.653752:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766121.653759:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766121.653764:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066372736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766121.653775:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766121.653802:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766121.653816:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766121/real 1775766121] req@ffff9d8190553480 x1862023066372864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766121.653830:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066372864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766121.653840:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766121.653844:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766121.653848:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766121.653979:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766121.654018:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766121.654039:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766121/real 1775766121] req@ffff9d8190553100 x1862023066372992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766121.654064:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066372992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766121.654093:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766121.654100:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766121.654106:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066372992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766121.654117:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766121.654146:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766121.654159:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766121/real 1775766121] req@ffff9d8190550e00 x1862023066373120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766121.654173:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066373120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766121.654184:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766121.654188:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766121.654191:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766121.654198:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766121.654215:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766121.654225:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766121/real 1775766121] req@ffff9d8190551500 x1862023066373248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766121.654237:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066373248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766121.654247:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766121.654250:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766121.654254:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766122.293198:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766122.676985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766123.317065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766123.701028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766124.341105:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766124.725036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766125.364995:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766125.749030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766126.389044:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766126.772969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766126.772984:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766126.772991:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766126.772994:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766126.773004:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2958 00000100:00080000:2.0:1775766126.773013:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766126.773018:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766126.773051:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066373376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766126.773075:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766126.773079:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766126.773081:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766126.773085:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2958 00000100:00080000:2.0:1775766126.773089:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766126.773094:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766126.773103:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766126.773117:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066373504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766126.773128:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766126.773129:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066373376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766126.773132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766126.773134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766126.773138:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2958 00000100:00080000:2.0:1775766126.773141:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766126.773144:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766126.773149:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766126.773158:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066373632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766126.773166:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766126.773169:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766126.773171:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766126.773175:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2958 00000100:00080000:2.0:1775766126.773189:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766126.773193:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766126.773197:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766126.773207:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066373760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766126.773216:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766126.773219:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766126.773222:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766126.773226:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2958 00000100:00080000:2.0:1775766126.773229:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766126.773233:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766126.773237:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766126.773244:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775766126.773248:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066373888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766126.773276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766126.773301:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766126.773322:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766127.413005:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766127.796989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766128.437019:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766128.820971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766129.461019:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766129.845036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775766129.845184:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775766129.845232:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766129.845247:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766129.845280:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766129.845287:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766129.845296:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766129.845318:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766129.845351:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766126/real 1775766129] req@ffff9d81984a2a00 x1862023066373888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766129.845381:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066373888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766129.845396:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766129.845402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766129.845407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373888:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766129.845417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766126/real 1775766129] req@ffff9d81984a3b80 x1862023066373504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766129.845431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066373504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766129.845445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766129.845449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766129.845454:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766129.845461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766126/real 1775766129] req@ffff9d81984a0700 x1862023066373632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766129.845474:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066373632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766129.845483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766129.845486:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766129.845489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766129.845495:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766126/real 1775766129] req@ffff9d81984a2d80 x1862023066373760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766129.845506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066373760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766129.845515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766129.845519:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766129.845521:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066373760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766129.845529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766126/real 1775766129] req@ffff9d81984a3480 x1862023066373376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766129.845540:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066373376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766129.845547:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766129.845551:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766129.845554:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066373376:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775766130.485044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766130.868966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766131.509044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766131.892980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766131.892994:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766131.893000:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766131.893003:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766131.893012:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2963 00000100:00080000:3.0:1775766131.893021:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766131.893026:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766131.893056:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066374016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766131.893078:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766131.893081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766131.893082:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766131.893084:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2963 00000100:00080000:3.0:1775766131.893086:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766131.893089:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766131.893097:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766131.893105:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066374144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766131.893113:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066374016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766131.893126:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766131.893128:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766131.893129:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766131.893133:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2963 00000100:00080000:3.0:1775766131.893135:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766131.893137:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766131.893140:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766131.893148:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066374272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766131.893154:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766131.893157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766131.893158:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766131.893160:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2963 00000100:00080000:3.0:1775766131.893170:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766131.893172:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766131.893178:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766131.893182:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766131.893193:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066374400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766131.893204:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766131.893207:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766131.893209:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766131.893214:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2963 00000100:00080000:3.0:1775766131.893219:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766131.893224:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766131.893229:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766131.893243:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766131/real 1775766131] req@ffff9d8188740000 x1862023066374016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766131.893243:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066374528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766131.893286:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066374016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766131.893299:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766131.893303:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766131.893308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066374016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766131.893330:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766131.893357:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766131.893371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766131/real 1775766131] req@ffff9d8188740380 x1862023066374144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766131.893383:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066374144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766131.893391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766131.893394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766131.893398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766131.893404:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766131.893425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766131.893436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766131/real 1775766131] req@ffff9d8188740700 x1862023066374272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766131.893447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066374272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766131.893455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766131.893459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766131.893463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766131.893468:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766131.893487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766131.893498:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766131/real 1775766131] req@ffff9d8188742a00 x1862023066374400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766131.893509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066374400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766131.893517:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766131.893520:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766131.893522:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766131.893528:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766131.893544:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766131.893553:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766131/real 1775766131] req@ffff9d8188741c00 x1862023066374528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766131.893565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066374528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766131.893575:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766131.893578:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766131.893580:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766132.533029:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766132.916968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766133.557158:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766133.940982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766134.581006:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766134.964939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766135.605221:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766135.988950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766136.629038:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766137.012993:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766137.012998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766137.013001:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766137.013007:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766137.013016:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2968 00000100:00080000:3.0:1775766137.013025:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766137.013031:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766137.013066:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066374656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766137.013089:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766137.013093:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766137.013094:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766137.013096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2968 00000100:00080000:3.0:1775766137.013099:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766137.013101:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766137.013108:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766137.013117:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066374784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766137.013125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766137.013128:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766137.013129:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766137.013134:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066374656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766137.013135:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2968 00000100:00080000:3.0:1775766137.013139:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766137.013146:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766137.013149:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766137.013156:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066374912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766137.013162:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766137.013165:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766137.013166:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766137.013168:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2968 00000100:00080000:3.0:1775766137.013172:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766137.013176:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766137.013181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766137.013192:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066375040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766137.013201:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775766137.013207:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766137.013208:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766137.013212:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766137.013218:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2968 00000100:00080000:3.0:1775766137.013223:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766137.013227:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766137.013232:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766137.013244:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066375168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766137.013272:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766137/real 1775766137] req@ffff9d8188741880 x1862023066374656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766137.013293:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066374656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766137.013304:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766137.013308:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766137.013312:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066374656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766137.013333:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766137.013355:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766137.013366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766137/real 1775766137] req@ffff9d8188742680 x1862023066374784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766137.013377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066374784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766137.013385:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766137.013388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766137.013392:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766137.013398:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766137.013416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766137.013427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766137/real 1775766137] req@ffff9d8188743800 x1862023066374912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766137.013437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066374912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766137.013444:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766137.013447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766137.013450:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066374912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766137.013455:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766137.013472:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766137.013484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766137/real 1775766137] req@ffff9d8188741f80 x1862023066375040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766137.013496:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066375040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766137.013505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766137.013508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766137.013511:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766137.013517:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766137.013532:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766137.013541:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766137/real 1775766137] req@ffff9d8188742300 x1862023066375168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766137.013551:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066375168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766137.013559:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766137.013563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766137.013566:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766137.653059:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766138.036957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766138.677050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766139.060967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766139.701031:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766140.084974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766140.725044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766141.109005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766141.749085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766142.132941:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766142.132949:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766142.132953:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766142.132955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766142.132961:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2973 00000100:00080000:3.0:1775766142.132971:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766142.132986:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766142.133019:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066375296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766142.133040:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766142.133043:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766142.133044:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766142.133047:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2973 00000100:00080000:3.0:1775766142.133049:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766142.133053:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766142.133060:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766142.133070:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066375296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766142.133071:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066375424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766142.133085:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766142.133087:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766142.133088:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766142.133091:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2973 00000100:00080000:3.0:1775766142.133093:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766142.133096:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766142.133099:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766142.133108:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066375552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766142.133114:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766142.133116:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766142.133118:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766142.133120:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2973 00000100:00080000:3.0:1775766142.133122:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766142.133124:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766142.133127:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766142.133129:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766142.133139:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066375680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766142.133145:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766142.133147:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766142.133149:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766142.133151:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2973 00000100:00080000:3.0:1775766142.133154:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766142.133157:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766142.133161:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766142.133170:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766142/real 1775766142] req@ffff9d8188741180 x1862023066375296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766142.133171:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066375808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766142.133213:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066375296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766142.133221:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766142.133225:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766142.133228:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066375296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766142.133249:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766142.133272:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766142.133285:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766142/real 1775766142] req@ffff9d8188743100 x1862023066375424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766142.133299:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066375424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766142.133308:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766142.133310:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766142.133312:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766142.133317:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766142.133332:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766142.133339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766142/real 1775766142] req@ffff9d8188740e00 x1862023066375552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766142.133352:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066375552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766142.133357:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766142.133359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766142.133361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766142.133364:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766142.133375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766142.133380:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766142/real 1775766142] req@ffff9d8188740a80 x1862023066375680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766142.133387:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066375680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766142.133393:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766142.133395:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766142.133396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766142.133400:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766142.133409:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766142.133414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766142/real 1775766142] req@ffff9d8188741500 x1862023066375808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766142.133421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066375808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766142.133430:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766142.133432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766142.133434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066375808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766142.773044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766143.156992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766143.797131:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766144.180999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766144.821010:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766145.204985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766145.845040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766146.228992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766146.869049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766147.253044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766147.253064:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766147.253074:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766147.253081:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766147.253094:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2978 00000100:00080000:1.0:1775766147.253108:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766147.253117:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766147.253163:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066375936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766147.253192:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766147.253196:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766147.253198:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766147.253202:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2978 00000100:00080000:1.0:1775766147.253206:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766147.253210:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766147.253221:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766147.253238:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066376064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766147.253239:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066375936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766147.253259:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766147.253262:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766147.253264:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766147.253267:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2978 00000100:00080000:1.0:1775766147.253271:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766147.253285:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766147.253290:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766147.253306:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066376192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766147.253316:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775766147.253317:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766147.253323:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766147.253326:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766147.253331:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2978 00000100:00080000:1.0:1775766147.253337:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766147.253340:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766147.253344:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766147.253356:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066376320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766147.253366:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766147.253369:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766147.253371:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766147.253377:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766147/real 1775766147] req@ffff9d8190551500 x1862023066375936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766147.253381:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2978 00000100:00080000:1.0:1775766147.253388:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766147.253397:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766147.253404:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766147.253414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066375936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766147.253419:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066376448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766147.253457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766147.253469:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766147.253476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066375936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766147.253500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766147.253521:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766147.253532:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766147/real 1775766147] req@ffff9d8190550e00 x1862023066376064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766147.253550:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066376064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766147.253558:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766147.253561:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766147.253564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766147.253568:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766147.253585:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766147.253598:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766147/real 1775766147] req@ffff9d8190553100 x1862023066376192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766147.253660:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066376192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766147.253674:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766147.253679:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766147.253684:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766147.253692:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766147.253715:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766147.253726:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766147/real 1775766147] req@ffff9d8190553480 x1862023066376320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766147.253741:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066376320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766147.253762:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766147.253767:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766147.253771:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766147.253780:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766147.253808:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766147.253823:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766147/real 1775766147] req@ffff9d8190551880 x1862023066376448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766147.253834:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066376448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766147.253842:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766147.253845:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766147.253848:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766147.893057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766148.277066:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766148.917002:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766149.300998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766149.941045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00020000:2.0:1775766150.324947:0:33225:0:(osc_cache.c:992:osc_extent_wait()) extent ffff9d819031d0c0@{[0 -> 0/1023], [3|0|+|rpc|wihY|ffff9d81a01b36c0], [28672|1|+|-|ffff9d818583a000|1024|ffff9d81980da7c0]} lustre-OST0001-osc-ffff9d81892c2000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:1.0:1775766150.324967:0:33224:0:(osc_cache.c:992:osc_extent_wait()) extent ffff9d819031d600@{[0 -> 76/1023], [3|0|+|rpc|wihY|ffff9d81a01b3340], [339968|77|+|-|ffff9d8190010800|1024|ffff9d81980d8000]} lustre-OST0000-osc-ffff9d81892c2000: wait ext to 0 timedout, recovery in progress? 00000008:00020000:1.0:1775766150.324979:0:33224:0:(osc_cache.c:992:osc_extent_wait()) ### extent: 00000000e9bb7054 ns: lustre-OST0000-osc-ffff9d81892c2000 lock: 000000000a26ef3c/0xeef44110eff57ea lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x228:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->315391) gid 0 flags: 0x800029400020000 nid: local remote: 0x4dcde3aa32155d21 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1775766150.325100:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000008:00020000:2.0:1775766150.337733:0:33225:0:(osc_cache.c:992:osc_extent_wait()) ### extent: 000000002b6030ac ns: lustre-OST0001-osc-ffff9d81892c2000 lock: 000000009c8bfad7/0xeef44110eff57c7 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x227:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800029400020000 nid: local remote: 0x4dcde3aa32155cc6 expref: -99 pid: 34105 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1775766150.965089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766151.348984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766151.989001:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766152.372970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766152.372979:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766152.372986:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766152.372990:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766152.372999:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2983 00000100:00080000:3.0:1775766152.373010:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766152.373016:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766152.373051:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066376576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766152.373078:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766152.373081:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766152.373083:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766152.373086:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2983 00000100:00080000:3.0:1775766152.373088:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766152.373092:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766152.373102:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766152.373112:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066376704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766152.373117:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066376576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766152.373133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766152.373136:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766152.373137:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766152.373141:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2983 00000100:00080000:3.0:1775766152.373143:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766152.373147:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766152.373150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766152.373160:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066376832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766152.373167:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766152.373169:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766152.373171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766152.373174:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2983 00000100:00080000:3.0:1775766152.373178:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766152.373180:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766152.373185:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766152.373187:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766152.373217:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066376960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766152.373227:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766152.373230:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766152.373232:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766152.373236:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2983 00000100:00080000:3.0:1775766152.373241:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766152.373245:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766152.373249:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766152.373260:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066377088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766152.373271:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766152/real 1775766152] req@ffff9d8188743b80 x1862023066376576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766152.373317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066376576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766152.373327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766152.373332:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766152.373336:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066376576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766152.373356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766152.373377:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766152.373388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766152/real 1775766152] req@ffff9d8188742d80 x1862023066376704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766152.373399:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066376704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766152.373406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766152.373409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766152.373412:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766152.373417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766152.373430:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766152.373438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766152/real 1775766152] req@ffff9d8188743480 x1862023066376832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766152.373447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066376832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766152.373460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766152.373463:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766152.373466:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766152.373470:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766152.373483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766152.373490:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766152/real 1775766152] req@ffff9d8188741500 x1862023066376960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766152.373499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066376960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766152.373507:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766152.373510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766152.373512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066376960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766152.373516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766152.373529:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766152.373536:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766152/real 1775766152] req@ffff9d8188740a80 x1862023066377088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766152.373545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066377088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766152.373552:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766152.373554:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766152.373557:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766153.013068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766153.397014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766154.037084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766154.420976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766155.061071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766155.444995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766156.085048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766156.468960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766157.109105:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766157.492970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766157.492988:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766157.492993:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766157.492997:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766157.493004:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2989 00000100:00080000:1.0:1775766157.493014:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766157.493020:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766157.493054:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066377216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766157.493076:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766157.493079:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766157.493080:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766157.493082:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2989 00000100:00080000:1.0:1775766157.493085:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766157.493087:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766157.493094:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766157.493101:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066377344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766157.493108:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766157.493110:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766157.493111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766157.493114:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2989 00000100:00080000:1.0:1775766157.493116:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766157.493118:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766157.493120:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766157.493125:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066377472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766157.493130:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766157.493132:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766157.493133:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766157.493136:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2989 00000100:00080000:1.0:1775766157.493138:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766157.493141:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766157.493144:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766157.493150:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066377600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766157.493157:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766157.493162:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775766157.493163:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066377216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766157.493164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766157.493171:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2989 00000100:00080000:1.0:1775766157.493173:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766157.493176:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766157.493179:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766157.493195:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066377728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766157.493223:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766157.493264:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766157/real 1775766157] req@ffff9d8190551880 x1862023066377216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766157.493279:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066377216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766157.493287:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766157.493289:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766157.493292:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066377216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766157.493297:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766157.493311:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766157.493318:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766157/real 1775766157] req@ffff9d8190553480 x1862023066377344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766157.493326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066377344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766157.493332:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766157.493334:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766157.493336:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766157.493340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766157.493350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766157.493355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766157/real 1775766157] req@ffff9d8190553100 x1862023066377472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766157.493362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066377472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766157.493367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766157.493369:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766157.493371:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766157.493389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766157.493399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766157.493405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766157/real 1775766157] req@ffff9d8190550e00 x1862023066377600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766157.493412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066377600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766157.493418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766157.493420:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766157.493422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766157.493425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766157.493434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766157.493439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766157/real 1775766157] req@ffff9d8190551500 x1862023066377728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766157.493446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066377728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766157.493451:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766157.493453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766157.493455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766158.133021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766158.517029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766159.157180:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766159.540959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766160.181103:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766160.565000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766161.205108:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766161.588982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766162.229102:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766162.613021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766162.613073:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766162.613083:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766162.613088:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766162.613098:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2994 00000100:00080000:2.0:1775766162.613112:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766162.613120:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766162.613168:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066377856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766162.613200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766162.613203:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766162.613205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766162.613209:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2994 00000100:00080000:2.0:1775766162.613211:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766162.613214:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766162.613224:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766162.613244:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066377984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766162.613254:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766162.613258:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066377856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766162.613259:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766162.613265:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766162.613288:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2994 00000100:00080000:2.0:1775766162.613291:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766162.613294:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766162.613298:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766162.613310:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066378112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766162.613317:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766162.613320:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766162.613322:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766162.613326:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2994 00000100:00080000:2.0:1775766162.613347:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766162.613350:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766162.613354:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766162.613361:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766162.613370:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066378240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766162.613380:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766162.613384:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766162.613386:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766162.613389:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2994 00000100:00080000:2.0:1775766162.613392:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766162.613394:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766162.613400:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766162.613415:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066378368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766162.613446:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766162/real 1775766162] req@ffff9d81984a0000 x1862023066377856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766162.613491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066377856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766162.613514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766162.613518:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766162.613522:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066377856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766162.613550:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766162.613576:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766162.613588:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766162/real 1775766162] req@ffff9d81984a2680 x1862023066377984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766162.613601:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066377984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766162.613610:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766162.613613:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766162.613617:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066377984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766162.613622:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766162.613638:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766162.613646:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766162/real 1775766162] req@ffff9d81984a2300 x1862023066378112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766162.613658:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066378112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766162.613677:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766162.613680:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766162.613683:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766162.613688:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766162.613704:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766162.613713:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766162/real 1775766162] req@ffff9d81984a1f80 x1862023066378240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766162.613725:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066378240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766162.613734:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766162.613737:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766162.613739:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766162.613745:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766162.613761:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766162.613775:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766162/real 1775766162] req@ffff9d81984a1180 x1862023066378368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766162.613786:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066378368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766162.613794:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766162.613797:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766162.613800:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766163.253027:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766163.637026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766164.277138:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766164.661000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766165.301140:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766165.684961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766166.325042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766166.708979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766167.349143:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766167.732942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766167.732961:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766167.732967:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766167.732971:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766167.732978:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 2999 00000100:00080000:3.0:1775766167.732987:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766167.732992:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766167.733022:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066378496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766167.733045:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766167.733050:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766167.733051:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766167.733054:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2999 00000100:00080000:3.0:1775766167.733056:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766167.733059:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766167.733067:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766167.733075:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066378496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766167.733079:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066378624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766167.733089:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766167.733091:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766167.733092:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766167.733096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2999 00000100:00080000:3.0:1775766167.733100:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766167.733104:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766167.733110:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766167.733123:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066378752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766167.733133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775766167.733137:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766167.733142:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766167.733144:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766167.733151:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2999 00000100:00080000:3.0:1775766167.733156:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766167.733161:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766167.733168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766167.733180:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766167/real 1775766167] req@ffff9d8188740e00 x1862023066378496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766167.733182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066378880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766167.733210:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766167.733213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766167.733214:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766167.733219:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066378496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766167.733219:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 2999 00000100:00080000:3.0:1775766167.733223:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766167.733230:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775766167.733235:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775766167.733235:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775766167.733243:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766167.733247:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066379008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766167.733248:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066378496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766167.733275:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766167.733291:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766167.733316:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766167/real 1775766167] req@ffff9d8188743100 x1862023066378624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766167.733328:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066378624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766167.733335:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766167.733338:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766167.733341:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766167.733345:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766167.733360:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766167.733384:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766167/real 1775766167] req@ffff9d8188741180 x1862023066378752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766167.733396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066378752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766167.733401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766167.733402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766167.733404:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766167.733408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766167.733437:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766167.733445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766167/real 1775766167] req@ffff9d8188742300 x1862023066378880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766167.733452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066378880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766167.733469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766167.733472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766167.733474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066378880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766167.733478:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766167.733488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766167.733493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766167/real 1775766167] req@ffff9d8188741f80 x1862023066379008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766167.733501:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066379008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766167.733511:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766167.733513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766167.733515:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766168.373090:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766168.757012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766169.397061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766169.780953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766170.421051:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766170.804958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766171.445007:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766171.829003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766172.469032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766172.852965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766172.853003:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766172.853012:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766172.853016:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766172.853024:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3004 00000100:00080000:3.0:1775766172.853034:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766172.853039:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766172.853091:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066379136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766172.853121:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766172.853125:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766172.853127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766172.853130:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3004 00000100:00080000:3.0:1775766172.853133:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766172.853137:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766172.853147:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766172.853158:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066379264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766172.853167:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066379136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766172.853173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766172.853181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766172.853183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766172.853187:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3004 00000100:00080000:3.0:1775766172.853189:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766172.853192:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766172.853196:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766172.853221:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066379392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766172.853227:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766172.853230:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766172.853231:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766172.853235:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3004 00000100:00080000:3.0:1775766172.853239:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766172.853243:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766172.853248:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766172.853254:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066379520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766172.853262:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766172.853264:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766172.853265:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766172.853267:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3004 00000100:00080000:3.0:1775766172.853269:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766172.853273:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766172.853282:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766172.853282:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766172.853291:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066379648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766172.853428:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766172/real 1775766172] req@ffff9d8188743800 x1862023066379136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.853448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066379136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.853457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.853461:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766172.853464:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066379136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766172.853489:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766172.853510:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766172.853529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766172/real 1775766172] req@ffff9d8188742680 x1862023066379264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.853544:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066379264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.853551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.853553:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766172.853555:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766172.853559:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766172.853571:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766172.853577:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766172/real 1775766172] req@ffff9d8188741880 x1862023066379392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.853585:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066379392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.853601:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.853603:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766172.853605:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766172.853609:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766172.853628:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766172.853636:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766172/real 1775766172] req@ffff9d8188741c00 x1862023066379520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.853643:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066379520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.853655:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.853657:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766172.853660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766172.853664:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766172.853676:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766172.853681:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766172/real 1775766172] req@ffff9d8188742a00 x1862023066379648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766172.853689:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066379648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766172.853695:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766172.853697:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766172.853699:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766173.493061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766173.876933:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766174.517204:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766174.900965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766175.541042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766175.925022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766176.565082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766176.948968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766177.589131:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766177.972989:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766177.972994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766177.972998:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766177.973004:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766177.973012:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3009 00000100:00080000:3.0:1775766177.973022:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766177.973028:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766177.973062:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066379776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766177.973085:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766177.973089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766177.973090:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766177.973093:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3009 00000100:00080000:3.0:1775766177.973095:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766177.973098:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766177.973105:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766177.973115:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066379904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766177.973116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066379776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766177.973129:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766177.973131:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766177.973132:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766177.973136:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3009 00000100:00080000:3.0:1775766177.973138:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766177.973139:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766177.973142:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766177.973152:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066380032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766177.973157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766177.973160:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766177.973161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766177.973163:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3009 00000100:00080000:3.0:1775766177.973167:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766177.973169:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766177.973174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766177.973178:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766177.973185:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066380160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766177.973192:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766177.973194:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766177.973195:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766177.973197:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3009 00000100:00080000:3.0:1775766177.973199:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766177.973201:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766177.973204:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766177.973212:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066380288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766177.973220:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766177/real 1775766177] req@ffff9d8188740700 x1862023066379776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766177.973240:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066379776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766177.973247:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766177.973250:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766177.973253:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066379776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766177.973269:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766177.973287:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766177.973299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766177/real 1775766177] req@ffff9d8188740380 x1862023066379904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766177.973311:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066379904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766177.973334:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766177.973338:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766177.973341:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066379904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766177.973347:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766177.973367:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766177.973375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766177/real 1775766177] req@ffff9d8188740000 x1862023066380032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766177.973383:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066380032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766177.973392:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766177.973394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766177.973396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766177.973400:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766177.973412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766177.973417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766177/real 1775766177] req@ffff9d8188742a00 x1862023066380160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766177.973424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066380160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766177.973430:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766177.973432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766177.973434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766177.973437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766177.973446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766177.973451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766177/real 1775766177] req@ffff9d8188741c00 x1862023066380288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766177.973458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066380288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766177.973463:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766177.973465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766177.973467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766178.613168:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766178.996954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766179.637084:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766180.020973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766180.661077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766181.044969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766181.685005:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766182.069028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766182.709024:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766183.092929:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766183.092937:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766183.092942:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766183.092961:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3014 00000100:00080000:3.0:1775766183.092970:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766183.092977:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766183.092991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775766183.093010:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066380416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766183.093044:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766183.093048:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766183.093050:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766183.093054:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3014 00000100:00080000:3.0:1775766183.093058:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766183.093062:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766183.093071:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766183.093084:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066380544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766183.093102:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066380416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766183.093104:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766183.093107:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766183.093109:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766183.093113:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3014 00000100:00080000:3.0:1775766183.093116:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766183.093119:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766183.093123:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766183.093133:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066380672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766183.093141:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766183.093144:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766183.093145:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766183.093149:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3014 00000100:00080000:3.0:1775766183.093151:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766183.093166:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766183.093170:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766183.093172:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766183.093179:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066380800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766183.093188:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766183.093190:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766183.093192:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766183.093195:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3014 00000100:00080000:3.0:1775766183.093198:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766183.093201:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766183.093205:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766183.093215:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066380928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766183.093233:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766183/real 1775766183] req@ffff9d8188741880 x1862023066380416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766183.093252:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066380416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766183.093263:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766183.093267:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766183.093282:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066380416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766183.093322:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766183.093348:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766183.093360:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766183/real 1775766183] req@ffff9d8188742680 x1862023066380544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766183.093369:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066380544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766183.093382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766183.093386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766183.093389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766183.093395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766183.093416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766183.093439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766183/real 1775766183] req@ffff9d8188743800 x1862023066380672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766183.093452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066380672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766183.093457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766183.093460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766183.093462:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766183.093473:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766183.093489:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766183.093496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766183/real 1775766183] req@ffff9d8188741f80 x1862023066380800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766183.093503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066380800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766183.093510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766183.093512:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766183.093514:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766183.093518:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766183.093528:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766183.093533:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766183/real 1775766183] req@ffff9d8188742300 x1862023066380928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766183.093541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066380928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766183.093546:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766183.093547:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766183.093549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066380928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766183.733063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766184.117039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766184.757096:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766185.140969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766185.781019:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766186.165064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766186.805086:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766187.188977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766187.829078:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766188.213037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766188.213047:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766188.213058:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766188.213064:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766188.213086:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3019 00000100:00080000:3.0:1775766188.213102:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766188.213111:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766188.213158:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066381056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766188.213208:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766188.213213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766188.213216:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766188.213282:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3019 00000100:00080000:3.0:1775766188.213291:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766188.213301:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766188.213317:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766188.213339:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066381056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766188.213365:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066381184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766188.213384:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766188.213389:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766188.213402:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766188.213408:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3019 00000100:00080000:3.0:1775766188.213432:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766188.213437:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766188.213446:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766188.213447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766188.213471:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066381312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766188.213484:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766188.213489:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766188.213492:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766188.213497:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3019 00000100:00080000:3.0:1775766188.213503:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766188.213508:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766188.213515:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766188.213520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766188/real 1775766188] req@ffff9d8188741180 x1862023066381056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766188.213536:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066381440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766188.213565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066381056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766188.213578:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766188.213597:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766188.213603:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766188.213603:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766188.213631:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775766188.213633:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3019 00000100:00080000:3.0:1775766188.213647:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766188.213649:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066381056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766188.213663:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766188.213685:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766188.213713:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066381568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766188.213715:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766188.213784:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766188.213807:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766188/real 1775766188] req@ffff9d8188743100 x1862023066381184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766188.213835:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066381184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766188.213845:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766188.213850:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766188.213854:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766188.213862:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766188.213925:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766188.213940:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766188/real 1775766188] req@ffff9d8188740e00 x1862023066381312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766188.213955:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066381312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766188.213965:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766188.213969:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766188.213973:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766188.213981:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766188.214002:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766188.214012:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766188/real 1775766188] req@ffff9d8188740a80 x1862023066381440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766188.214035:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066381440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766188.214046:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766188.214050:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766188.214053:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766188.214080:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766188.214099:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766188.214109:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766188/real 1775766188] req@ffff9d8188741500 x1862023066381568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766188.214123:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066381568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766188.214133:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766188.214137:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766188.214140:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766188.853061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766189.237016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766189.877030:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766190.261024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766190.901233:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766191.285032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766191.925085:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766192.309023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766192.949106:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766193.333049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766193.333086:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766193.333094:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766193.333100:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766193.333109:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3024 00000100:00080000:2.0:1775766193.333119:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766193.333126:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766193.333165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066381696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766193.333234:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766193.333238:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766193.333240:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766193.333244:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3024 00000100:00080000:2.0:1775766193.333248:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766193.333253:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766193.333263:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766193.333280:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066381824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766193.333283:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066381696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766193.333303:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766193.333306:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766193.333307:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766193.333312:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3024 00000100:00080000:2.0:1775766193.333316:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766193.333319:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766193.333323:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766193.333335:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066381952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766193.333343:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766193.333346:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766193.333348:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766193.333350:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3024 00000100:00080000:2.0:1775766193.333357:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766193.333360:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766193.333363:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766193.333373:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066382080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766193.333381:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766193.333384:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766193.333386:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766193.333389:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3024 00000100:00080000:2.0:1775766193.333392:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766193.333397:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766193.333405:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766193.333424:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066382208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766193.333439:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766193.333479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766193.333502:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766193.333532:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766193.973042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766194.356990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766194.997096:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766195.381001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766196.021071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766196.405033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775766196.405382:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775766196.405443:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766196.405465:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766196.405516:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766196.405526:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766196.405535:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766196.405546:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766196.405605:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766193/real 1775766196] req@ffff9d81984a0e00 x1862023066382208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766196.405665:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066382208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766196.405679:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766196.405685:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766196.405690:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382208:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766196.405700:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766193/real 1775766196] req@ffff9d81984a1880 x1862023066381824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766196.405717:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066381824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766196.405728:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766196.405732:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766196.405736:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766196.405744:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766193/real 1775766196] req@ffff9d81984a3800 x1862023066381952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766196.405756:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066381952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766196.405765:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766196.405769:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766196.405773:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066381952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766196.405781:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766193/real 1775766196] req@ffff9d81984a3100 x1862023066382080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766196.405794:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066382080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766196.405805:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766196.405808:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766196.405812:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766196.405820:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766193/real 1775766196] req@ffff9d81984a1500 x1862023066381696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766196.405835:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066381696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766196.405843:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766196.405847:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766196.405850:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066381696:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775766197.045042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766197.429013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766198.069075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766198.452988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766198.453026:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766198.453033:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766198.453036:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766198.453045:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3029 00000100:00080000:1.0:1775766198.453054:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766198.453060:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766198.453106:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066382336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766198.453145:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766198.453148:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766198.453150:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766198.453154:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3029 00000100:00080000:1.0:1775766198.453157:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766198.453160:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766198.453169:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766198.453181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066382464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766198.453190:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766198.453193:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766198.453195:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766198.453199:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3029 00000100:00080000:1.0:1775766198.453201:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766198.453207:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066382336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766198.453207:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766198.453215:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766198.453223:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066382592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766198.453232:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766198.453235:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766198.453237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766198.453240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3029 00000100:00080000:1.0:1775766198.453243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766198.453245:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766198.453249:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766198.453258:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066382720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766198.453269:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766198.453272:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766198.453275:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766198.453279:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3029 00000800:00000100:0.0:1775766198.453286:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766198.453287:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766198.453295:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766198.453302:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766198.453316:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066382848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766198.453350:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766198/real 1775766198] req@ffff9d8190551500 x1862023066382336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766198.453379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066382336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766198.453390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766198.453396:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766198.453400:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066382336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766198.453408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766198.453436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766198.453451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766198/real 1775766198] req@ffff9d8190550e00 x1862023066382464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766198.453481:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066382464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766198.453491:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766198.453495:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766198.453498:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766198.453522:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766198.453544:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766198.453556:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766198/real 1775766198] req@ffff9d8190553100 x1862023066382592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766198.453580:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066382592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766198.453601:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766198.453606:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766198.453610:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766198.453616:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766198.453638:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766198.453649:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766198/real 1775766198] req@ffff9d8190553480 x1862023066382720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766198.453663:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066382720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766198.453674:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766198.453678:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766198.453682:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766198.453688:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766198.453709:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766198.453721:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766198/real 1775766198] req@ffff9d8190551880 x1862023066382848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766198.453733:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066382848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766198.453741:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766198.453744:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766198.453748:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066382848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766199.093176:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766199.476956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766200.117077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766200.500964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766201.141188:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766201.524968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766202.165208:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766202.548960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766203.189126:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766203.573001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766203.573071:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766203.573142:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766203.573146:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766203.573157:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3035 00000100:00080000:3.0:1775766203.573170:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766203.573178:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766203.573230:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066382976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766203.573266:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766203.573271:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766203.573272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766203.573276:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3035 00000100:00080000:3.0:1775766203.573280:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766203.573284:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766203.573293:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766203.573307:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066383104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766203.573314:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066382976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766203.573320:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766203.573324:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766203.573326:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766203.573330:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3035 00000100:00080000:3.0:1775766203.573333:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766203.573336:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766203.573341:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766203.573352:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066383232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766203.573360:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766203.573363:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766203.573365:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766203.573368:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3035 00000100:00080000:3.0:1775766203.573370:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766203.573373:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766203.573378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766203.573378:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766203.573391:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066383360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766203.573418:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766203.573421:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766203.573423:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766203.573426:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3035 00000100:00080000:3.0:1775766203.573429:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766203.573433:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766203.573437:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766203.573454:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066383488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766203.573475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766203/real 1775766203] req@ffff9d8188743480 x1862023066382976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766203.573512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066382976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766203.573521:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766203.573525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766203.573529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066382976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766203.573548:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766203.573568:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766203.573577:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766203/real 1775766203] req@ffff9d8188742d80 x1862023066383104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766203.573598:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066383104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766203.573605:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766203.573608:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766203.573611:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766203.573616:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766203.573628:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766203.573635:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766203/real 1775766203] req@ffff9d8188743b80 x1862023066383232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766203.573644:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066383232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766203.573651:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766203.573653:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766203.573655:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766203.573659:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766203.573672:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766203.573678:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766203/real 1775766203] req@ffff9d8188741500 x1862023066383360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766203.573687:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066383360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766203.573695:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766203.573697:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766203.573699:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766203.573703:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766203.573714:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766203.573721:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766203/real 1775766203] req@ffff9d8188740a80 x1862023066383488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766203.573730:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066383488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766203.573737:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766203.573739:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766203.573741:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766204.213168:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766204.597005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766205.237132:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766205.621013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766206.261225:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766206.644994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766207.285170:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766207.669001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766208.309158:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766208.693022:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766208.693045:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775766208.693050:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766208.693050:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766208.693063:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3040 00000100:00080000:3.0:1775766208.693086:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766208.693094:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766208.693131:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066383616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766208.693169:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766208.693173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766208.693175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766208.693180:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3040 00000100:00080000:3.0:1775766208.693184:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766208.693190:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766208.693201:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766208.693204:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066383616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766208.693217:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066383744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766208.693231:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766208.693235:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766208.693237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766208.693241:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3040 00000100:00080000:3.0:1775766208.693255:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766208.693260:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766208.693264:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766208.693270:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766208.693286:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066383872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766208.693297:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766208.693301:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766208.693303:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766208.693308:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3040 00000100:00080000:3.0:1775766208.693313:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766208.693314:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766208/real 1775766208] req@ffff9d8188740e00 x1862023066383616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766208.693320:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766208.693329:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766208.693350:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066383616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766208.693351:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066384000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766208.693384:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766208.693386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766208.693389:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766208.693392:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766208.693394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775766208.693402:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3040 00000100:00100000:0.0:1775766208.693403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066383616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766208.693407:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766208.693414:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766208.693420:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766208.693433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766208.693434:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066384128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766208.693493:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766208.693520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766208/real 1775766208] req@ffff9d8188743100 x1862023066383744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766208.693562:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066383744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766208.693575:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766208.693580:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766208.693585:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766208.693593:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766208.693618:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766208.693644:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766208/real 1775766208] req@ffff9d8188741180 x1862023066383872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766208.693656:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066383872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766208.693664:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766208.693667:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766208.693670:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066383872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766208.693687:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766208.693704:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766208.693714:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766208/real 1775766208] req@ffff9d8188742300 x1862023066384000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766208.693727:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066384000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766208.693765:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766208.693771:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766208.693776:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766208.693802:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766208.693845:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766208.693860:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766208/real 1775766208] req@ffff9d8188741f80 x1862023066384128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766208.693890:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066384128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766208.693898:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766208.693901:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766208.693904:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766209.333250:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766209.717107:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766210.357030:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766210.740968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766211.381070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766211.764982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766212.405049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766212.789051:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766213.429137:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766213.813020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766213.813060:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766213.813086:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766213.813092:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766213.813103:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3045 00000100:00080000:1.0:1775766213.813114:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766213.813121:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766213.813160:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066384256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766213.813208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766213.813211:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766213.813213:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766213.813216:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3045 00000100:00080000:1.0:1775766213.813219:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766213.813222:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766213.813230:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766213.813240:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066384384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766213.813255:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766213.813258:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766213.813260:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766213.813264:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3045 00000100:00080000:1.0:1775766213.813266:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766213.813273:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766213.813276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066384256:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775766213.813278:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766213.813292:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066384512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766213.813301:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766213.813305:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766213.813306:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766213.813309:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3045 00000100:00080000:1.0:1775766213.813312:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766213.813314:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766213.813318:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766213.813327:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066384640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766213.813334:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766213.813337:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766213.813338:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766213.813342:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3045 00000800:00000100:0.0:1775766213.813350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766213.813350:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766213.813358:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766213.813361:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766213.813371:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066384768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766213.813417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766213/real 1775766213] req@ffff9d8190551880 x1862023066384256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766213.813442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066384256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766213.813463:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766213.813468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766213.813474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066384256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766213.813484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766213.813530:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766213.813544:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766213/real 1775766213] req@ffff9d8190553480 x1862023066384384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766213.813556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066384384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766213.813564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766213.813567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766213.813570:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766213.813592:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766213.813608:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766213.813616:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766213/real 1775766213] req@ffff9d8190553100 x1862023066384512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766213.813626:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066384512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766213.813633:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766213.813636:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766213.813639:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766213.813644:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766213.813658:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766213.813665:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766213/real 1775766213] req@ffff9d8190550e00 x1862023066384640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766213.813675:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066384640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766213.813683:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766213.813685:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766213.813688:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766213.813692:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766213.813705:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766213.813711:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766213/real 1775766213] req@ffff9d8190551500 x1862023066384768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766213.813721:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066384768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766213.813728:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766213.813730:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766213.813733:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066384768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766214.453036:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766214.837192:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766215.477132:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766215.861009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766216.501108:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766216.884991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766217.525286:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766217.909018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766218.549098:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766218.933064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766218.933066:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766218.933078:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766218.933083:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766218.933093:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3050 00000100:00080000:1.0:1775766218.933105:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766218.933113:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766218.933159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066384896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766218.933195:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766218.933200:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766218.933202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766218.933206:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3050 00000100:00080000:1.0:1775766218.933209:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766218.933213:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766218.933222:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766218.933255:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066385024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766218.933269:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766218.933277:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766218.933279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766218.933285:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3050 00000100:00080000:1.0:1775766218.933290:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766218.933299:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066384896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766218.933299:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766218.933306:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766218.933324:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066385152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766218.933337:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766218.933341:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766218.933343:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766218.933348:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3050 00000100:00080000:1.0:1775766218.933352:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766218.933355:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766218.933361:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766218.933373:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066385280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766218.933383:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766218.933387:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766218.933389:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766218.933392:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3050 00000100:00080000:1.0:1775766218.933398:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766218.933403:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766218.933417:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766218.933442:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766218.933457:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066385408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766218.933487:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766218/real 1775766218] req@ffff9d8190551500 x1862023066384896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766218.933530:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066384896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766218.933543:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766218.933548:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766218.933553:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066384896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766218.933562:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766218.933588:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766218.933601:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766218/real 1775766218] req@ffff9d8190550e00 x1862023066385024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766218.933615:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066385024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766218.933641:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766218.933644:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766218.933647:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766218.933671:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766218.933689:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766218.933699:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766218/real 1775766218] req@ffff9d8190553100 x1862023066385152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766218.933711:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066385152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766218.933719:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766218.933722:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766218.933725:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766218.933730:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766218.933745:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766218.933754:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766218/real 1775766218] req@ffff9d8190553480 x1862023066385280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766218.933765:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066385280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766218.933774:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766218.933776:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766218.933779:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766218.933784:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766218.933797:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766218.933805:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766218/real 1775766218] req@ffff9d8190551880 x1862023066385408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766218.933816:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066385408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766218.933825:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766218.933828:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766218.933831:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766219.573110:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766219.957039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766220.597231:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766220.981038:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766221.621136:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766222.004958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766222.645049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766223.028976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766223.669064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766224.052967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766224.052991:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766224.052995:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766224.052998:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766224.053006:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3055 00000100:00080000:3.0:1775766224.053015:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766224.053021:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766224.053071:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066385536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766224.053105:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766224.053110:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766224.053112:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766224.053118:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3055 00000100:00080000:3.0:1775766224.053122:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766224.053128:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766224.053139:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766224.053159:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066385536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766224.053159:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066385664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766224.053178:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766224.053181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766224.053183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766224.053187:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3055 00000100:00080000:3.0:1775766224.053191:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766224.053195:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766224.053201:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766224.053211:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066385792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766224.053219:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766224.053222:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766224.053223:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766224.053246:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3055 00000800:00000100:0.0:1775766224.053247:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766224.053250:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766224.053254:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766224.053258:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766224.053267:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066385920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766224.053274:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766224.053276:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766224.053278:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766224.053280:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3055 00000100:00080000:3.0:1775766224.053282:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766224.053285:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766224.053289:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766224.053297:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066386048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766224.053305:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766224/real 1775766224] req@ffff9d8188743800 x1862023066385536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766224.053329:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066385536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766224.053342:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766224.053347:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766224.053352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066385536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766224.053374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766224.053397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766224.053412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766224/real 1775766224] req@ffff9d8188742680 x1862023066385664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766224.053426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066385664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766224.053443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766224.053447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766224.053451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766224.053458:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766224.053482:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766224.053494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766224/real 1775766224] req@ffff9d8188741880 x1862023066385792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766224.053508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066385792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766224.053519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766224.053522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766224.053526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766224.053533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766224.053551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766224.053560:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766224/real 1775766224] req@ffff9d8188741c00 x1862023066385920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766224.053573:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066385920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766224.053581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766224.053583:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766224.053602:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066385920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766224.053607:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766224.053623:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766224.053630:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766224/real 1775766224] req@ffff9d8188742a00 x1862023066386048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766224.053639:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066386048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766224.053646:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766224.053648:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766224.053651:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766224.693135:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766225.076943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766225.717140:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766226.100940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766226.741159:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766227.124981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766227.765088:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766228.148989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766228.789194:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766229.172969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766229.173013:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766229.173025:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766229.173031:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766229.173044:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3060 00000100:00080000:2.0:1775766229.173060:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766229.173068:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766229.173118:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066386176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766229.173154:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766229.173159:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766229.173161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766229.173166:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3060 00000100:00080000:2.0:1775766229.173170:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766229.173176:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766229.173189:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766229.173205:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066386176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766229.173213:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066386304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766229.173228:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766229.173233:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766229.173235:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766229.173243:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3060 00000100:00080000:2.0:1775766229.173249:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766229.173254:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766229.173261:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766229.173279:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066386432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766229.173280:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766229.173295:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766229.173300:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766229.173302:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766229.173307:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3060 00000100:00080000:2.0:1775766229.173313:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766229.173318:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766229.173323:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766229.173342:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766229/real 1775766229] req@ffff9d81984a0a80 x1862023066386176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766229.173343:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066386560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766229.173412:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766229.173418:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066386176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766229.173421:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766229.173428:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766229.173440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766229.173445:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3060 00000100:00080000:0.0:1775766229.173449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766229.173459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066386176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766229.173459:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766229.173472:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766229.173480:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766229.173496:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066386688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766229.173498:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766229.173530:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766229.173544:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766229/real 1775766229] req@ffff9d81984a1500 x1862023066386304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766229.173553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066386304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766229.173561:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766229.173563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766229.173566:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766229.173571:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766229.173582:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766229.173587:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766229/real 1775766229] req@ffff9d81984a3100 x1862023066386432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766229.173594:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066386432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766229.173600:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766229.173602:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766229.173603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766229.173607:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766229.173617:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766229.173623:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766229/real 1775766229] req@ffff9d81984a3800 x1862023066386560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766229.173630:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066386560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766229.173636:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766229.173638:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766229.173640:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766229.173649:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766229.173658:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766229.173664:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766229/real 1775766229] req@ffff9d81984a1880 x1862023066386688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766229.173670:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066386688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766229.173679:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766229.173682:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766229.173685:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766229.813103:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766230.196988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766230.837075:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766231.220958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766231.861062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766232.244975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766232.885029:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766233.268934:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766233.909052:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766234.292991:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766234.292993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766234.293001:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766234.293005:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766234.293013:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3065 00000100:00080000:2.0:1775766234.293022:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766234.293029:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766234.293061:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066386816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766234.293084:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766234.293087:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766234.293088:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766234.293090:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3065 00000100:00080000:2.0:1775766234.293092:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766234.293094:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766234.293101:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766234.293110:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066386944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766234.293117:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766234.293119:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766234.293121:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766234.293126:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3065 00000100:00100000:0.0:1775766234.293127:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066386816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766234.293128:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766234.293138:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766234.293145:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766234.293153:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066387072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766234.293161:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766234.293163:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766234.293164:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766234.293167:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3065 00000100:00080000:2.0:1775766234.293175:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766234.293177:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766234.293181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766234.293182:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766234.293192:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066387200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766234.293200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766234.293203:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766234.293204:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766234.293206:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3065 00000100:00080000:2.0:1775766234.293208:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766234.293211:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766234.293214:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766234.293222:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066387328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766234.293226:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766234/real 1775766234] req@ffff9d81984a0e00 x1862023066386816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766234.293248:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066386816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766234.293256:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766234.293260:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766234.293263:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066386816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766234.293268:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766234.293282:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766234.293289:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766234/real 1775766234] req@ffff9d81984a1180 x1862023066386944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766234.293297:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066386944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766234.293303:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766234.293305:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766234.293307:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066386944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766234.293323:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766234.293333:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766234.293339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766234/real 1775766234] req@ffff9d81984a1f80 x1862023066387072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766234.293346:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066387072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766234.293355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766234.293357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766234.293359:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766234.293362:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766234.293372:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766234.293377:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766234/real 1775766234] req@ffff9d81984a2300 x1862023066387200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766234.293384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066387200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766234.293390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766234.293392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766234.293394:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766234.293397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766234.293407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766234.293412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766234/real 1775766234] req@ffff9d81984a2680 x1862023066387328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766234.293419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066387328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766234.293423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766234.293425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766234.293428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766234.933082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766235.316938:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766235.956997:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766236.340997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766236.981087:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766237.364967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766238.005030:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766238.389027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766239.029065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766239.412950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766239.412969:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766239.412975:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766239.413006:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766239.413014:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3070 00000100:00080000:2.0:1775766239.413024:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766239.413029:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766239.413064:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066387456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766239.413088:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766239.413092:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766239.413093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766239.413096:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3070 00000100:00080000:2.0:1775766239.413099:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766239.413101:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766239.413108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766239.413121:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066387584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766239.413133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766239.413138:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766239.413140:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766239.413146:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3070 00000100:00100000:0.0:1775766239.413152:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066387456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766239.413152:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766239.413161:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766239.413169:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766239.413182:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066387712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766239.413191:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766239.413194:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766239.413195:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766239.413198:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3070 00000100:00080000:2.0:1775766239.413212:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766239.413220:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766239.413222:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775766239.413227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766239.413252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066387840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766239.413265:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766239.413269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766239.413271:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766239.413276:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3070 00000100:00080000:2.0:1775766239.413282:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766239.413284:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766239/real 1775766239] req@ffff9d81984a0000 x1862023066387456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766239.413289:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766239.413298:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766239.413322:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066387968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766239.413323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066387456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766239.413353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766239.413357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766239.413362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066387456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766239.413373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766239.413412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766239.413427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766239/real 1775766239] req@ffff9d81984a3480 x1862023066387584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766239.413439:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066387584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766239.413445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766239.413448:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766239.413450:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766239.413471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766239.413488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766239.413495:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766239/real 1775766239] req@ffff9d81984a2d80 x1862023066387712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766239.413504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066387712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766239.413520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766239.413522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766239.413524:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766239.413528:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766239.413539:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766239.413545:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766239/real 1775766239] req@ffff9d81984a0700 x1862023066387840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766239.413553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066387840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766239.413559:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766239.413561:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766239.413563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766239.413567:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766239.413579:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766239.413584:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766239/real 1775766239] req@ffff9d81984a3b80 x1862023066387968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766239.413593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066387968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766239.413599:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766239.413601:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766239.413603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066387968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766240.053100:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766240.436976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766241.077074:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766241.460959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766242.101019:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766242.484963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766243.125028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766243.508998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766244.149025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766244.533081:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766244.533089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766244.533104:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766244.533109:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766244.533131:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3076 00000100:00080000:2.0:1775766244.533153:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766244.533178:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766244.533229:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066388096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766244.533273:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766244.533278:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766244.533280:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766244.533286:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3076 00000100:00080000:2.0:1775766244.533291:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766244.533298:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766244.533323:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766244.533333:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066388096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766244.533343:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066388224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766244.533354:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766244.533356:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766244.533358:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766244.533362:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3076 00000100:00080000:2.0:1775766244.533367:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766244.533372:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766244.533378:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766244.533391:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066388352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766244.533409:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766244.533414:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766244.533416:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766244.533421:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3076 00000100:00080000:2.0:1775766244.533425:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766244.533430:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766244.533450:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766244.533463:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066388480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766244.533471:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766244.533474:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766244.533476:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766244.533478:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3076 00000100:00080000:2.0:1775766244.533482:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766244.533485:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766244.533489:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766244.533502:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066388608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766244.533503:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766244.533600:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766244/real 1775766244] req@ffff9d81984a2a00 x1862023066388096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766244.533620:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066388096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766244.533630:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766244.533635:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766244.533640:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066388096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766244.533699:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766244.533734:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766244.533744:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766244/real 1775766244] req@ffff9d81984a3b80 x1862023066388224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766244.533759:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066388224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766244.533765:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766244.533767:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766244.533769:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766244.533785:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766244.533797:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766244.533803:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766244/real 1775766244] req@ffff9d81984a0700 x1862023066388352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766244.533811:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066388352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766244.533817:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766244.533819:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766244.533821:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766244.533824:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766244.533836:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766244.533853:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766244/real 1775766244] req@ffff9d81984a2d80 x1862023066388480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766244.533860:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066388480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766244.533871:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766244.533896:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766244.533899:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766244.533904:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766244.533928:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766244.533935:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766244/real 1775766244] req@ffff9d81984a3480 x1862023066388608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766244.533942:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066388608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766244.533949:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766244.533951:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766244.533953:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766245.173099:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766245.556974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766246.197217:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766246.580981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766247.221160:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766247.604954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766248.245052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766248.629028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766249.269089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766249.652971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766249.652992:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766249.652998:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766249.653002:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766249.653009:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3081 00000100:00080000:1.0:1775766249.653018:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766249.653035:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766249.653070:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066388736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766249.653109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766249.653114:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766249.653116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766249.653120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3081 00000100:00080000:1.0:1775766249.653123:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766249.653127:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766249.653138:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766249.653163:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066388864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766249.653175:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766249.653178:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766249.653180:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766249.653184:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3081 00000100:00080000:1.0:1775766249.653188:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766249.653191:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766249.653195:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766249.653204:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066388992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766249.653213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766249.653216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766249.653218:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766249.653221:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3081 00000100:00080000:1.0:1775766249.653223:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766249.653231:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766249.653235:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066388736:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775766249.653235:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766249.653253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066389120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766249.653259:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766249.653263:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766249.653266:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766249.653271:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3081 00000100:00080000:1.0:1775766249.653274:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766249.653278:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766249.653283:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766249.653295:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066389248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766249.653436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766249.653494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766249/real 1775766249] req@ffff9d8190551880 x1862023066388736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766249.653516:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066388736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766249.653535:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766249.653540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766249.653544:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066388736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766249.653555:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766249.653584:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766249.653599:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766249/real 1775766249] req@ffff9d8190553480 x1862023066388864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766249.653614:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066388864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766249.653623:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766249.653627:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766249.653631:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766249.653678:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766249.653702:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766249.653715:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766249/real 1775766249] req@ffff9d8190553100 x1862023066388992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766249.653738:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066388992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766249.653748:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766249.653752:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766249.653756:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066388992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766249.653762:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766249.653786:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766249.653798:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766249/real 1775766249] req@ffff9d8190550e00 x1862023066389120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766249.653821:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066389120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766249.653832:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766249.653836:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766249.653840:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766249.653846:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766249.653868:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766249.653932:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766249/real 1775766249] req@ffff9d8190551500 x1862023066389248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766249.653948:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066389248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766249.653969:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766249.653973:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766249.653976:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766250.293123:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766250.677006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766251.317022:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766251.700974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766252.341069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766252.724980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766253.365057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766253.749020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766254.389171:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766254.772992:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766254.773001:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775766254.773002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766254.773006:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766254.773019:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3086 00000100:00080000:3.0:1775766254.773030:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766254.773038:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766254.773093:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066389376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766254.773126:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766254.773146:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766254.773152:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766254.773155:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3086 00000100:00080000:3.0:1775766254.773159:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766254.773163:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766254.773171:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766254.773186:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066389504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766254.773192:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066389376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766254.773223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766254.773227:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766254.773229:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766254.773235:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3086 00000100:00080000:3.0:1775766254.773239:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766254.773244:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766254.773249:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775766254.773266:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766254.773267:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066389632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766254.773279:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766254.773282:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766254.773284:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766254.773287:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3086 00000100:00080000:3.0:1775766254.773292:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766254.773296:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766254.773301:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766254.773315:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066389760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766254.773326:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766254.773328:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766254.773330:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766254.773333:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3086 00000100:00080000:3.0:1775766254.773337:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766254.773339:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766254.773343:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766254.773357:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766254/real 1775766254] req@ffff9d8188740000 x1862023066389376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766254.773357:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066389888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766254.773378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066389376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766254.773387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766254.773391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766254.773394:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066389376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766254.773414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766254.773431:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766254.773439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766254/real 1775766254] req@ffff9d8188740380 x1862023066389504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766254.773447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066389504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766254.773453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766254.773455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766254.773458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766254.773461:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766254.773471:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766254.773477:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766254/real 1775766254] req@ffff9d8188740700 x1862023066389632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766254.773485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066389632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766254.773490:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766254.773492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766254.773494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766254.773498:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766254.773508:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766254.773513:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766254/real 1775766254] req@ffff9d8188742a00 x1862023066389760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766254.773520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066389760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766254.773526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766254.773528:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766254.773530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766254.773533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766254.773542:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766254.773547:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766254/real 1775766254] req@ffff9d8188741c00 x1862023066389888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766254.773554:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066389888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766254.773564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766254.773566:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766254.773568:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066389888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766255.413065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766255.797099:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766256.437013:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766256.820993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766257.461024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766257.845002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766258.485143:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766258.869002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766259.509085:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766259.893036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766259.893036:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766259.893048:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766259.893054:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766259.893066:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3091 00000100:00080000:3.0:1775766259.893079:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766259.893087:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766259.893161:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066390016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766259.893215:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766259.893220:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766259.893223:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766259.893229:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3091 00000100:00080000:3.0:1775766259.893234:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766259.893240:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766259.893253:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766259.893274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066390144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766259.893281:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066390016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766259.893306:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766259.893312:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766259.893315:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766259.893324:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3091 00000100:00080000:3.0:1775766259.893346:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766259.893354:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766259.893362:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766259.893384:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066390272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766259.893404:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766259.893406:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390144:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775766259.893419:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766259.893423:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766259.893432:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3091 00000100:00080000:3.0:1775766259.893440:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766259.893447:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766259.893458:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766259.893462:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766259.893508:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066390400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766259.893533:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766259.893595:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766259.893599:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766259.893606:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3091 00000100:00080000:3.0:1775766259.893614:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766259.893621:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766259.893626:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390400:192.168.202.105@tcp:8:kworker.0 00000008:00100000:3.0:1775766259.893631:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766259.893658:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066390528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766259.893691:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766260.533048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766260.917059:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766261.557079:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766261.941026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766262.581165:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766262.964962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775766262.965144:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775766262.965190:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766262.965210:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766262.965253:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766262.965264:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766262.965270:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766262.965277:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766262.965332:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766259/real 1775766262] req@ffff9d8188742300 x1862023066390528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766262.965366:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066390528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766262.965378:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766262.965383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766262.965386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390528:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766262.965398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766259/real 1775766262] req@ffff9d8188741f80 x1862023066390400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766262.965412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066390400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766262.965422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766262.965426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766262.965430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766262.965439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766259/real 1775766262] req@ffff9d8188743800 x1862023066390272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766262.965453:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066390272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766262.965461:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766262.965464:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766262.965467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766262.965476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766259/real 1775766262] req@ffff9d8188742680 x1862023066390144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766262.965500:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066390144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766262.965523:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766262.965527:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766262.965531:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766262.965540:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766259/real 1775766262] req@ffff9d8188741880 x1862023066390016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766262.965556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066390016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766262.965563:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766262.965567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766262.965570:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066390016:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775766263.605078:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766263.988969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766264.629104:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766265.013022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766265.013049:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766265.013062:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766265.013069:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766265.013083:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3096 00000100:00080000:0.0:1775766265.013097:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766265.013106:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766265.013152:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066390656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766265.013205:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766265.013210:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766265.013213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766265.013217:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3096 00000100:00080000:0.0:1775766265.013222:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766265.013226:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766265.013239:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766265.013255:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066390784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766265.013269:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766265.013274:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766265.013277:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766265.013282:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3096 00000100:00080000:0.0:1775766265.013287:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766265.013291:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066390656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766265.013295:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766265.013310:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766265.013337:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066390912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766265.013352:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766265.013357:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766265.013359:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766265.013364:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3096 00000100:00080000:0.0:1775766265.013381:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766265.013396:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766265.013403:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766265.013419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766265.013421:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066391040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766265.013436:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766265.013442:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766265.013446:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766265.013450:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3096 00000100:00080000:0.0:1775766265.013455:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766265.013459:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766265.013466:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766265.013484:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066391168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766265.013529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766265/real 1775766265] req@ffff9d81901af100 x1862023066390656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766265.013566:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066390656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766265.013581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766265.013586:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766265.013591:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066390656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766265.013602:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766265.013643:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766265.013666:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766265/real 1775766265] req@ffff9d81901afb80 x1862023066390784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766265.013692:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066390784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766265.013710:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766265.013716:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766265.013724:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766265.013766:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766265.013836:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766265.013859:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766265/real 1775766265] req@ffff9d81901aed80 x1862023066390912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766265.013961:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066390912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766265.013986:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766265.014183:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766265.014193:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066390912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766265.014207:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766265.014257:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766265.014279:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766265/real 1775766265] req@ffff9d81901ae680 x1862023066391040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766265.014305:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066391040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766265.014324:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766265.014331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766265.014338:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766265.014351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766265.014390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766265.014413:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766265/real 1775766265] req@ffff9d81901ad180 x1862023066391168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766265.014438:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066391168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766265.014455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766265.014461:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766265.014467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766265.653113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766266.037013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766266.677024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766267.061022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766267.701204:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766268.085027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766268.725122:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766269.109050:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766269.749061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766270.133050:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766270.133065:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766270.133073:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766270.133108:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3101 00000100:00100000:3.0:1775766270.133111:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766270.133125:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766270.133139:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766270.133200:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066391296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766270.133251:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766270.133257:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766270.133261:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766270.133270:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3101 00000100:00080000:0.0:1775766270.133278:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766270.133287:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766270.133302:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766270.133306:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066391296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766270.133341:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066391424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766270.133365:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766270.133373:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766270.133376:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766270.133385:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3101 00000100:00080000:0.0:1775766270.133409:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766270.133419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766270.133421:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766270.133436:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766270.133493:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066391552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766270.133515:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766270.133523:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766270.133527:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766270.133537:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766270/real 1775766270] req@ffff9d81901ad180 x1862023066391296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766270.133545:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3101 00000100:00080000:0.0:1775766270.133562:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766270.133574:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766270.133581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066391296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766270.133597:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1775766270.133603:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:3.0:1775766270.133608:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766270.133619:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066391296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766270.133639:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066391680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766270.133659:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766270.133665:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766270.133670:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766270.133672:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391424:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775766270.133691:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3101 00000100:00080000:0.0:1775766270.133700:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766270.133708:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766270.133719:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775766270.133729:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766270.133751:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066391808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766270.133770:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766270/real 1775766270] req@ffff9d81901ae680 x1862023066391424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766270.133793:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066391424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766270.133804:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766270.133814:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766270.133818:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766270.133826:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766270.133849:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766270.133859:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766270/real 1775766270] req@ffff9d81901aed80 x1862023066391552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766270.133871:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066391552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766270.133910:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766270.133913:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766270.133916:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766270.133923:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766270.133944:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766270.133955:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766270/real 1775766270] req@ffff9d81901afb80 x1862023066391680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766270.133967:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066391680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766270.133977:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766270.133981:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766270.133984:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766270.134003:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766270.134033:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766270.134051:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766270/real 1775766270] req@ffff9d81901af100 x1862023066391808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766270.134072:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066391808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766270.134087:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766270.134093:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766270.134098:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066391808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766270.773086:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766271.157022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766271.797110:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766272.180944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766272.821076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766273.205032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766273.845118:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775766274.228974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766274.869033:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766275.253011:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766275.253022:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775766275.253023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766275.253028:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766275.253039:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3106 00000100:00080000:3.0:1775766275.253053:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766275.253061:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766275.253107:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066391936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766275.253150:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766275.253154:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766275.253156:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766275.253159:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3106 00000100:00080000:3.0:1775766275.253161:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766275.253164:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766275.253175:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766275.253184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066392064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766275.253193:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766275.253196:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766275.253198:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766275.253202:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066391936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766275.253208:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3106 00000100:00080000:3.0:1775766275.253215:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766275.253218:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766275.253221:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766275.253237:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066392192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766275.253243:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766275.253246:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766275.253248:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766275.253252:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3106 00000100:00080000:3.0:1775766275.253287:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766275.253290:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766275.253293:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766275.253301:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766275.253312:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066392320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766275.253320:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766275.253323:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766275.253324:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766275.253327:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3106 00000100:00080000:3.0:1775766275.253330:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766275.253343:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766275.253347:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766275.253357:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066392448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766275.253386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766275/real 1775766275] req@ffff9d8188741180 x1862023066391936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766275.253423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066391936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766275.253433:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766275.253438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766275.253442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066391936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766275.253466:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766275.253490:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766275.253511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766275/real 1775766275] req@ffff9d8188743100 x1862023066392064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766275.253528:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066392064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766275.253535:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766275.253538:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766275.253541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766275.253546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766275.253561:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766275.253568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766275/real 1775766275] req@ffff9d8188740e00 x1862023066392192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766275.253577:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066392192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766275.253586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766275.253590:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766275.253593:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766275.253601:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766275.253624:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766275.253652:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766275/real 1775766275] req@ffff9d8188740a80 x1862023066392320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766275.253665:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066392320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766275.253681:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766275.253685:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766275.253688:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766275.253695:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766275.253731:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766275.253744:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766275/real 1775766275] req@ffff9d8188741500 x1862023066392448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766275.253754:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066392448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766275.253760:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766275.253763:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766275.253765:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766275.893108:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766276.276936:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766276.917028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766277.301081:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766277.941098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766278.324975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766278.965033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766279.348972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766279.989061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766280.373015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766280.373060:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766280.373067:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766280.373073:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766280.373085:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3111 00000100:00080000:0.0:1775766280.373098:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766280.373105:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766280.373151:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066392576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766280.373182:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766280.373185:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766280.373186:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766280.373189:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3111 00000100:00080000:0.0:1775766280.373192:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766280.373195:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766280.373205:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766280.373214:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066392704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766280.373223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766280.373226:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766280.373228:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766280.373236:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3111 00000100:00100000:3.0:1775766280.373236:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066392576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766280.373245:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766280.373257:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766280.373261:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766280.373272:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066392832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766280.373284:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766280.373287:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766280.373289:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766280.373291:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3111 00000100:00080000:0.0:1775766280.373304:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766280.373307:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766280.373310:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766280.373317:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766280.373327:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066392960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766280.373335:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766280.373338:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766280.373362:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766280.373366:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3111 00000100:00080000:0.0:1775766280.373371:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766280.373374:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766280.373380:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766280.373395:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066393088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766280.373398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766280/real 1775766280] req@ffff9d81901af100 x1862023066392576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766280.373458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066392576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766280.373469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766280.373472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766280.373475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066392576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766280.373496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766280.373522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766280.373530:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766280/real 1775766280] req@ffff9d81901afb80 x1862023066392704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766280.373539:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066392704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766280.373545:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766280.373547:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766280.373549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766280.373554:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766280.373564:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766280.373570:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766280/real 1775766280] req@ffff9d81901aed80 x1862023066392832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766280.373578:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066392832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766280.373583:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766280.373585:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766280.373587:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766280.373590:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766280.373600:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766280.373606:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766280/real 1775766280] req@ffff9d81901ae680 x1862023066392960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766280.373617:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066392960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766280.373623:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766280.373625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766280.373627:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066392960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766280.373630:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766280.373640:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766280.373645:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766280/real 1775766280] req@ffff9d81901ad180 x1862023066393088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766280.373652:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066393088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766280.373658:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766280.373660:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766280.373661:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766281.013154:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766281.397002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766282.037117:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766282.421054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766283.061153:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766283.445013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766284.085038:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766284.468962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766285.109042:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766285.492997:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766285.493009:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766285.493015:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766285.493026:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3117 00000100:00100000:2.0:1775766285.493036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766285.493039:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766285.493056:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766285.493107:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066393216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766285.493134:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766285.493138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766285.493139:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766285.493142:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3117 00000100:00080000:0.0:1775766285.493145:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766285.493148:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766285.493159:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766285.493171:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066393344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766285.493182:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766285.493190:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766285.493190:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066393216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766285.493198:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766285.493214:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3117 00000100:00080000:0.0:1775766285.493217:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766285.493220:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766285.493226:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766285.493239:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066393472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766285.493245:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766285.493248:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766285.493249:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766285.493253:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3117 00000100:00080000:0.0:1775766285.493264:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766285.493267:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775766285.493272:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775766285.493273:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766285.493342:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766285/real 1775766285] req@ffff9d81901ad180 x1862023066393216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766285.493357:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066393216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766285.493367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766285.493371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766285.493374:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066393216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766285.493394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766285.493412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766285.493420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766285/real 1775766285] req@ffff9d81901ae680 x1862023066393344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766285.493428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066393344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766285.493434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766285.493438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766285.493439:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066393600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766285.493443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393344:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775766285.493466:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766285.493469:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393472:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775766285.493471:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766285.493475:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766285.493483:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3117 00000100:00080000:0.0:1775766285.493489:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766285.493493:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766285.493498:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775766285.493511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766285.493515:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066393728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766285.493529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766285/real 1775766285] req@ffff9d81901aed80 x1862023066393472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766285.493540:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066393472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766285.493547:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766285.493550:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766285.493553:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766285.493566:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766285.493580:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766285.493587:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766285/real 1775766285] req@ffff9d81901afb80 x1862023066393600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766285.493595:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066393600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766285.493601:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766285.493603:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766285.493604:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766285.493608:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766285.493621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766285.493631:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766285/real 1775766285] req@ffff9d81901af100 x1862023066393728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766285.493642:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066393728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766285.493652:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766285.493656:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766285.493659:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766286.133072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766286.516953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766287.157080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766287.540947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766288.181148:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766288.564979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766289.205169:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766289.588982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766290.229106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766290.612988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766290.612994:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766290.612999:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766290.613003:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766290.613012:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3122 00000100:00080000:3.0:1775766290.613020:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766290.613026:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766290.613059:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066393856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766290.613083:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766290.613086:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766290.613087:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766290.613090:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3122 00000100:00080000:3.0:1775766290.613092:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766290.613095:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766290.613103:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766290.613112:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066393984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766290.613116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066393856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766290.613124:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766290.613127:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766290.613128:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766290.613131:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3122 00000100:00080000:3.0:1775766290.613133:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766290.613135:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766290.613139:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766290.613148:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066394112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766290.613155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766290.613157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766290.613158:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766290.613160:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3122 00000100:00080000:3.0:1775766290.613169:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766290.613171:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766290.613174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766290.613176:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766290.613186:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066394240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766290.613193:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766290.613195:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766290.613196:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766290.613199:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3122 00000100:00080000:3.0:1775766290.613201:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766290.613204:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766290.613207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766290.613219:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066394368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766290.613220:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766290/real 1775766290] req@ffff9d8188743b80 x1862023066393856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766290.613269:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066393856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766290.613278:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766290.613281:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766290.613285:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066393856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766290.613304:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766290.613321:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766290.613330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766290/real 1775766290] req@ffff9d8188742d80 x1862023066393984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766290.613338:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066393984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766290.613343:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766290.613345:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766290.613348:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066393984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766290.613351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766290.613361:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766290.613366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766290/real 1775766290] req@ffff9d8188743480 x1862023066394112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766290.613374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066394112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766290.613379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766290.613380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766290.613382:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766290.613385:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766290.613396:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766290.613402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766290/real 1775766290] req@ffff9d8188741500 x1862023066394240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766290.613409:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066394240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766290.613414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766290.613416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766290.613418:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766290.613421:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766290.613431:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766290.613436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766290/real 1775766290] req@ffff9d8188740a80 x1862023066394368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766290.613443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066394368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766290.613453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766290.613455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766290.613457:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766291.253084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766291.637012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766292.277076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766292.661054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766293.301080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766293.684973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766294.325097:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766294.709039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766295.349259:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766295.732996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766295.733040:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766295.733051:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766295.733056:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766295.733069:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3127 00000100:00080000:1.0:1775766295.733083:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766295.733092:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766295.733142:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066394496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766295.733186:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766295.733191:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766295.733213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766295.733219:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3127 00000100:00080000:1.0:1775766295.733227:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766295.733235:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766295.733260:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766295.733263:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066394496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766295.733287:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066394624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766295.733306:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766295.733312:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766295.733314:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766295.733320:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3127 00000100:00080000:1.0:1775766295.733326:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766295.733331:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766295.733339:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775766295.733340:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766295.733362:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066394752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766295.733377:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766295.733382:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766295.733385:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766295.733391:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3127 00000100:00080000:1.0:1775766295.733397:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766295.733404:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766295.733409:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766295/real 1775766295] req@ffff9d8190551500 x1862023066394496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775766295.733414:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766295.733440:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066394880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766295.733442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066394496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766295.733455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766295.733462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766295.733472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066394496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766295.733492:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766295.733498:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766295.733501:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766295.733507:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394624:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775766295.733514:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3127 00000100:00080000:1.0:1775766295.733524:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766295.733531:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766295.733541:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766295.733546:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766295.733563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766295/real 1775766295] req@ffff9d8190550e00 x1862023066394624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766295.733571:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066395008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766295.733587:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066394624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766295.733606:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766295.733611:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766295.733617:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766295.733637:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766295.733656:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766295.733665:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766295/real 1775766295] req@ffff9d8190553100 x1862023066394752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766295.733677:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066394752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766295.733685:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766295.733688:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766295.733691:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766295.733697:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766295.733713:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766295.733721:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766295/real 1775766295] req@ffff9d8190553480 x1862023066394880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766295.733732:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066394880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766295.733741:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766295.733744:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766295.733747:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066394880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766295.733761:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766295.733775:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766295.733783:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766295/real 1775766295] req@ffff9d8190551880 x1862023066395008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766295.733793:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066395008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766295.733801:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766295.733804:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766295.733806:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766296.373088:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766296.757028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766297.397158:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766297.780966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766298.421026:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766298.805058:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766299.445083:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766299.828970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766300.469099:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766300.852951:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766300.852958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766300.852961:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766300.852966:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766300.852974:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3132 00000100:00080000:1.0:1775766300.852983:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766300.852989:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766300.853022:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066395136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766300.853055:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766300.853058:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766300.853059:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766300.853062:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3132 00000100:00080000:1.0:1775766300.853064:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766300.853067:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766300.853074:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766300.853085:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066395136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766300.853085:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066395264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766300.853097:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766300.853101:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766300.853103:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766300.853106:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3132 00000100:00080000:1.0:1775766300.853108:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766300.853110:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766300.853122:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766300.853135:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066395392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766300.853162:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775766300.853167:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766300.853170:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766300.853173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766300.853180:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3132 00000100:00080000:1.0:1775766300.853187:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766300.853191:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766300.853198:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766300.853214:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066395520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766300.853216:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766300/real 1775766300] req@ffff9d8190551880 x1862023066395136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766300.853236:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766300.853240:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766300.853241:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766300.853245:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3132 00000100:00100000:0.0:1775766300.853248:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066395136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766300.853249:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766300.853259:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775766300.853262:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775766300.853266:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775766300.853267:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766300.853273:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066395136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766300.853282:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066395648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766300.853293:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766300.853313:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766300.853325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766300/real 1775766300] req@ffff9d8190553480 x1862023066395264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766300.853333:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066395264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766300.853339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766300.853350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766300.853352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766300.853356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766300.853368:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766300.853374:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766300/real 1775766300] req@ffff9d8190553100 x1862023066395392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766300.853382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066395392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766300.853391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766300.853393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766300.853406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766300.853409:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766300.853419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766300.853424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766300/real 1775766300] req@ffff9d8190550e00 x1862023066395520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766300.853431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066395520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766300.853437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766300.853438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766300.853440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766300.853449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766300.853459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766300.853464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766300/real 1775766300] req@ffff9d8190551500 x1862023066395648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766300.853472:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066395648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766300.853477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766300.853479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766300.853481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766301.493006:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766301.876974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766302.517172:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766302.901014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766303.541026:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766303.924972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766304.565179:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766304.948956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766305.589096:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766305.972968:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766305.972979:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766305.972985:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766305.972989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766305.972996:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3137 00000100:00080000:0.0:1775766305.973008:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766305.973016:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766305.973080:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066395776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766305.973158:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766305.973163:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766305.973166:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766305.973172:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3137 00000100:00080000:0.0:1775766305.973178:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766305.973200:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766305.973211:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766305.973227:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066395904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766305.973244:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766305.973246:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066395776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766305.973247:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766305.973251:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766305.973258:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3137 00000100:00080000:0.0:1775766305.973262:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766305.973266:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766305.973272:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766305.973287:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066396032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766305.973299:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766305.973304:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775766305.973305:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766305.973309:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766305.973314:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3137 00000100:00080000:0.0:1775766305.973318:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766305.973323:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766305.973327:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766305.973338:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066396160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766305.973346:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766305.973350:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775766305.973350:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766305/real 1775766305] req@ffff9d81901af100 x1862023066395776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766305.973354:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766305.973362:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3137 00000100:00080000:0.0:1775766305.973368:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766305.973372:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766305.973376:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766305.973379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066395776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766305.973387:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066396288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766305.973389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766305.973399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766305.973403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066395776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766305.973423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766305.973440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766305.973448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766305/real 1775766305] req@ffff9d81901afb80 x1862023066395904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766305.973457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066395904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766305.973463:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766305.973465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766305.973468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066395904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766305.973472:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766305.973482:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766305.973487:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766305/real 1775766305] req@ffff9d81901aed80 x1862023066396032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766305.973495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066396032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766305.973501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766305.973503:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766305.973505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766305.973508:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766305.973519:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766305.973524:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766305/real 1775766305] req@ffff9d81901ae680 x1862023066396160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766305.973532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066396160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766305.973538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766305.973540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766305.973541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766305.973545:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766305.973555:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766305.973560:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766305/real 1775766305] req@ffff9d81901ad180 x1862023066396288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766305.973567:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066396288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766305.973578:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766305.973580:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766305.973582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766306.613150:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766306.997082:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766307.637141:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766308.021021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766308.661181:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766309.045027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766309.685115:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766310.069030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766310.709115:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766311.093009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766311.093017:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766311.093026:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766311.093031:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766311.093075:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3142 00000100:00080000:1.0:1775766311.093090:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766311.093099:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766311.093145:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066396416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766311.093171:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766311.093174:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766311.093175:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766311.093179:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3142 00000100:00080000:1.0:1775766311.093182:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766311.093185:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766311.093195:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766311.093204:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066396544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766311.093212:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766311.093216:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766311.093218:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766311.093218:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066396416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766311.093225:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3142 00000100:00080000:1.0:1775766311.093229:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766311.093231:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766311.093234:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766311.093243:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066396672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766311.093249:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766311.093251:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766311.093253:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766311.093256:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3142 00000100:00080000:1.0:1775766311.093259:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766311.093261:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766311.093265:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766311.093271:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066396800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766311.093277:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766311.093280:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766311.093281:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766311.093283:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3142 00000100:00080000:1.0:1775766311.093286:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766311.093288:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766311.093291:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775766311.093293:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766311.093301:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066396928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766311.093354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766311/real 1775766311] req@ffff9d8190551500 x1862023066396416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766311.093379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066396416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766311.093388:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766311.093392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766311.093395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066396416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766311.093417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766311.093434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766311.093442:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766311/real 1775766311] req@ffff9d8190550e00 x1862023066396544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766311.093451:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066396544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766311.093457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766311.093459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766311.093462:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766311.093473:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766311.093486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766311.093492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766311/real 1775766311] req@ffff9d8190553100 x1862023066396672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766311.093500:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066396672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766311.093506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766311.093508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766311.093510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766311.093514:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766311.093524:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766311.093530:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766311/real 1775766311] req@ffff9d8190553480 x1862023066396800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766311.093538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066396800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766311.093544:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766311.093545:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766311.093547:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766311.093550:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766311.093560:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766311.093566:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766311/real 1775766311] req@ffff9d8190551880 x1862023066396928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766311.093573:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066396928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766311.093579:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766311.093580:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766311.093582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066396928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766311.733091:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766312.117220:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766312.757124:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766313.140967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766313.781127:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766314.165091:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766314.805236:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766315.189003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766315.829130:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766316.213090:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766316.213102:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766316.213105:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766316.213113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766316.213135:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3147 00000100:00080000:1.0:1775766316.213152:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766316.213164:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766316.213226:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066397056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766316.213280:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766316.213285:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766316.213288:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766316.213292:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3147 00000100:00080000:1.0:1775766316.213297:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766316.213301:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766316.213314:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766316.213340:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066397184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766316.213353:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766316.213358:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766316.213362:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766316.213366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066397056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766316.213371:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3147 00000100:00080000:1.0:1775766316.213379:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766316.213384:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766316.213390:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766316.213406:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066397312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766316.213416:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766316.213419:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766316.213421:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766316.213426:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3147 00000100:00080000:1.0:1775766316.213430:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766316.213433:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766316.213439:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766316.213450:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066397440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766316.213458:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766316.213462:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766316.213465:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766316.213469:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3147 00000100:00080000:1.0:1775766316.213473:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766316.213476:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766316.213498:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775766316.213501:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766316.213514:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066397568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766316.213596:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766316/real 1775766316] req@ffff9d8190551880 x1862023066397056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766316.213623:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066397056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766316.213664:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766316.213672:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766316.213678:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066397056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766316.213717:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766316.213776:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766316.213792:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766316/real 1775766316] req@ffff9d8190553480 x1862023066397184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766316.213831:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066397184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766316.213840:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766316.213843:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766316.213847:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766316.213853:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766316.213874:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766316.213883:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766316/real 1775766316] req@ffff9d8190553100 x1862023066397312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766316.213894:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066397312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766316.213965:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766316.213969:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766316.213973:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766316.213980:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766316.214055:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766316.214069:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766316/real 1775766316] req@ffff9d8190550e00 x1862023066397440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766316.214082:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066397440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766316.214099:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766316.214120:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766316.214123:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766316.214130:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766316.214151:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766316.214160:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766316/real 1775766316] req@ffff9d8190551500 x1862023066397568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766316.214172:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066397568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766316.214181:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766316.214183:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766316.214186:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766316.853133:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766317.236949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766317.877110:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766318.261052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766318.901148:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766319.285064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766319.925107:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766320.309095:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766320.949137:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766321.333075:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766321.333078:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766321.333093:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766321.333100:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766321.333113:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3152 00000100:00080000:1.0:1775766321.333127:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766321.333137:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766321.333188:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066397696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766321.333231:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766321.333236:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766321.333238:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766321.333243:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3152 00000100:00080000:1.0:1775766321.333248:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766321.333252:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766321.333263:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766321.333277:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066397824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766321.333290:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766321.333295:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766321.333297:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766321.333304:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3152 00000100:00080000:1.0:1775766321.333308:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766321.333312:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766321.333317:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766321.333330:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066397952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766321.333338:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066397696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766321.333366:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766321.333370:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766321.333372:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766321.333379:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3152 00000100:00080000:1.0:1775766321.333382:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766321.333386:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766321.333393:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766321.333407:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066398080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766321.333419:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766321.333423:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766321.333425:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766321.333429:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3152 00000100:00080000:1.0:1775766321.333435:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766321.333441:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766321.333448:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766321.333455:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766321.333470:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066398208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766321.333520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766321/real 1775766321] req@ffff9d8190551500 x1862023066397696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766321.333546:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066397696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766321.333560:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766321.333567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766321.333572:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066397696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766321.333582:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766321.333609:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766321.333622:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766321/real 1775766321] req@ffff9d8190550e00 x1862023066397824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766321.333638:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066397824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766321.333647:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766321.333651:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766321.333655:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766321.333704:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766321.333728:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766321.333740:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766321/real 1775766321] req@ffff9d8190553100 x1862023066397952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766321.333755:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066397952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766321.333765:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766321.333769:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766321.333773:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066397952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766321.333779:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766321.333799:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766321.333809:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766321/real 1775766321] req@ffff9d8190553480 x1862023066398080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766321.333823:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066398080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766321.333833:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766321.333837:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766321.333840:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766321.333846:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766321.333863:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766321.333913:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766321/real 1775766321] req@ffff9d8190551880 x1862023066398208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766321.333931:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066398208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766321.333970:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766321.333976:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766321.333980:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766321.973200:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766322.357049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766322.997098:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766323.381068:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766324.021240:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766324.405015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766325.045073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766325.429077:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766326.069270:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766326.453023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766326.453039:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766326.453046:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766326.453051:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766326.453061:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3157 00000100:00080000:2.0:1775766326.453072:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766326.453079:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766326.453124:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066398336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766326.453152:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766326.453155:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766326.453156:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766326.453159:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3157 00000100:00080000:2.0:1775766326.453162:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766326.453165:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766326.453175:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766326.453184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066398464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766326.453192:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766326.453195:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766326.453197:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766326.453200:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3157 00000100:00080000:2.0:1775766326.453202:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766326.453204:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766326.453207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766326.453218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066398592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766326.453224:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066398336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766326.453232:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766326.453235:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766326.453236:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766326.453240:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3157 00000100:00080000:2.0:1775766326.453245:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766326.453248:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766326.453252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766326.453261:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066398720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766326.453272:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766326.453275:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766326.453277:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766326.453280:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3157 00000100:00080000:2.0:1775766326.453298:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766326.453300:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766326.453303:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766326.453311:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066398848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766326.453357:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766326.453399:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766326.453418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766326.453433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766327.093073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766327.477002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766328.117143:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766328.501008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766329.141095:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766329.525054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775766329.525292:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775766329.525363:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766329.525389:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766329.525449:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766329.525463:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766329.525472:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766329.525480:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775766329.525565:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766326/real 1775766329] req@ffff9d81984a2300 x1862023066398592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766329.525622:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066398592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766329.525644:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766329.525652:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766329.525661:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766329.525675:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766326/real 1775766329] req@ffff9d81984a1f80 x1862023066398720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766329.525698:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066398720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766329.525716:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766329.525724:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766329.525730:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766329.525744:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766326/real 1775766329] req@ffff9d81984a1180 x1862023066398848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766329.525766:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066398848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766329.525783:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766329.525790:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766329.525796:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398848:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766329.525807:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766326/real 1775766329] req@ffff9d81984a0000 x1862023066398336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766329.525829:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066398336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766329.525844:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766329.525851:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766329.525856:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066398336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766329.525867:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766326/real 1775766329] req@ffff9d81984a2680 x1862023066398464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766329.525980:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066398464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766329.525996:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766329.526003:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766329.526010:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066398464:192.168.202.105@tcp:38:kworker.0 00010000:00010000:3.0:1775766330.165170:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766330.548962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766331.189033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766331.573026:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766331.573029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766331.573038:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766331.573044:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766331.573056:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3163 00000100:00080000:2.0:1775766331.573068:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766331.573076:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766331.573132:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066398976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766331.573160:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766331.573163:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766331.573165:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766331.573168:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3163 00000100:00080000:2.0:1775766331.573170:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766331.573172:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766331.573182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766331.573194:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066399104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766331.573202:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766331.573204:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766331.573206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766331.573209:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3163 00000100:00080000:2.0:1775766331.573211:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766331.573213:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766331.573216:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766331.573227:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066399232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766331.573239:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766331.573240:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066398976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766331.573244:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766331.573248:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766331.573256:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3163 00000100:00080000:2.0:1775766331.573259:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766331.573262:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766331.573265:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766331.573275:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066399360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766331.573282:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766331.573284:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766331.573286:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766331.573289:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3163 00000100:00080000:2.0:1775766331.573291:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766331.573294:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766331.573297:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766331.573303:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066399488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766331.573354:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766331.573406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766331/real 1775766331] req@ffff9d81984a0e00 x1862023066398976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766331.573426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066398976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766331.573434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766331.573438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766331.573442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066398976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766331.573451:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766331.573481:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766331.573495:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766331/real 1775766331] req@ffff9d81984a1880 x1862023066399104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766331.573507:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066399104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766331.573516:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766331.573520:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766331.573523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766331.573550:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766331.573578:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766331.573591:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766331/real 1775766331] req@ffff9d81984a3800 x1862023066399232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766331.573604:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066399232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766331.573616:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766331.573619:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766331.573622:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766331.573628:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766331.573647:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766331.573656:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766331/real 1775766331] req@ffff9d81984a3100 x1862023066399360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766331.573667:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066399360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766331.573676:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766331.573679:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766331.573688:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766331.573695:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766331.573714:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766331.573725:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766331/real 1775766331] req@ffff9d81984a1500 x1862023066399488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766331.573736:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066399488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766331.573743:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766331.573746:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766331.573750:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766332.213114:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766332.596995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766333.237099:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766333.621011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766334.261111:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766334.644928:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766335.285049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766335.668998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766336.309085:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766336.692988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766336.692996:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766336.693007:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766336.693013:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766336.693025:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3168 00000100:00080000:2.0:1775766336.693039:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766336.693047:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766336.693118:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066399616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766336.693153:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766336.693156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766336.693158:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766336.693162:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3168 00000100:00080000:2.0:1775766336.693181:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766336.693184:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766336.693195:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766336.693210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066399744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766336.693242:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766336.693243:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066399616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766336.693251:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766336.693254:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766336.693264:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3168 00000100:00080000:2.0:1775766336.693267:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766336.693270:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766336.693299:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766336.693313:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066399872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766336.693322:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766336.693326:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766336.693327:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766336.693332:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3168 00000100:00080000:2.0:1775766336.693336:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766336.693339:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766336.693344:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766336.693352:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766336.693362:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066400000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766336.693375:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766336.693379:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766336.693380:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766336.693384:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3168 00000100:00080000:2.0:1775766336.693388:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766336.693394:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766336.693400:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766336.693415:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066400128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766336.693425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766336/real 1775766336] req@ffff9d81984a0a80 x1862023066399616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766336.693461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066399616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766336.693475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766336.693481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766336.693486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066399616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766336.693541:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766336.693567:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766336.693581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766336/real 1775766336] req@ffff9d81984a1500 x1862023066399744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766336.693596:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066399744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766336.693617:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766336.693621:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766336.693624:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766336.693633:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766336.693654:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766336.693666:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766336/real 1775766336] req@ffff9d81984a3100 x1862023066399872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766336.693680:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066399872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766336.693689:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766336.693694:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766336.693697:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066399872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766336.693705:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766336.693726:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766336.693738:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766336/real 1775766336] req@ffff9d81984a3800 x1862023066400000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766336.693752:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066400000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766336.693774:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766336.693778:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766336.693781:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766336.693789:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766336.693828:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766336.693856:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766336/real 1775766336] req@ffff9d81984a1880 x1862023066400128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766336.693878:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066400128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766336.693889:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766336.693919:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766336.693924:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766337.333052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766337.716951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766338.357081:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766338.740979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766339.381046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766339.764958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766340.404997:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766340.788982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766341.429055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766341.812981:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766341.812987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766341.812989:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766341.812994:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766341.813002:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3173 00000100:00080000:2.0:1775766341.813011:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766341.813016:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766341.813050:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066400256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766341.813072:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766341.813075:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766341.813076:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766341.813079:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3173 00000100:00080000:2.0:1775766341.813081:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766341.813083:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766341.813091:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766341.813099:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066400384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766341.813107:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766341.813111:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766341.813113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766341.813117:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3173 00000100:00080000:2.0:1775766341.813119:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766341.813121:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766341.813124:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766341.813130:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066400512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766341.813137:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766341.813139:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766341.813140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766341.813145:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3173 00000100:00080000:2.0:1775766341.813151:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766341.813153:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066400256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766341.813156:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766341.813168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766341.813177:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066400640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766341.813184:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766341.813188:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766341.813189:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766341.813192:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3173 00000100:00080000:2.0:1775766341.813194:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766341.813196:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766341.813199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766341.813206:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066400768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766341.813247:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766341.813299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766341/real 1775766341] req@ffff9d81984a0e00 x1862023066400256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766341.813316:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066400256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766341.813325:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766341.813329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766341.813332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066400256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766341.813339:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766341.813356:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766341.813364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766341/real 1775766341] req@ffff9d81984a2680 x1862023066400384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766341.813373:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066400384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766341.813384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766341.813386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766341.813389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766341.813412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766341.813424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766341.813430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766341/real 1775766341] req@ffff9d81984a0000 x1862023066400512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766341.813437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066400512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766341.813443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766341.813445:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766341.813447:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766341.813451:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766341.813463:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766341.813468:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766341/real 1775766341] req@ffff9d81984a1180 x1862023066400640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766341.813476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066400640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766341.813482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766341.813484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766341.813487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766341.813491:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766341.813501:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766341.813507:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766341/real 1775766341] req@ffff9d81984a1f80 x1862023066400768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766341.813514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066400768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766341.813520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766341.813522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766341.813523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066400768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766342.453052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766342.836970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766343.477074:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766343.860987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766344.501020:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766344.884952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766345.525007:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766345.908983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766346.549000:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766346.932934:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766346.932943:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766346.932963:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766346.932971:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3178 00000100:00080000:2.0:1775766346.932981:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766346.932987:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766346.932988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:2.0:1775766346.933040:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066400896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766346.933067:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766346.933070:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766346.933071:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766346.933074:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3178 00000100:00080000:2.0:1775766346.933077:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766346.933080:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766346.933087:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766346.933096:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066401024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766346.933103:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766346.933106:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766346.933107:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766346.933110:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3178 00000100:00080000:2.0:1775766346.933113:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766346.933115:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766346.933118:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766346.933125:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066401152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766346.933132:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766346.933135:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766346.933136:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766346.933139:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3178 00000100:00080000:2.0:1775766346.933142:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766346.933145:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766346.933148:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766346.933156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066401280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766346.933163:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066400896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766346.933168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766346.933188:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766346.933189:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766346.933192:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3178 00000100:00080000:2.0:1775766346.933195:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766346.933197:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766346.933200:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766346.933209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066401408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766346.933239:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766346.933278:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766346/real 1775766346] req@ffff9d81984a2300 x1862023066400896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766346.933292:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066400896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766346.933300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766346.933303:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766346.933306:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066400896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766346.933312:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766346.933326:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766346.933333:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766346/real 1775766346] req@ffff9d81984a3480 x1862023066401024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766346.933341:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066401024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766346.933347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766346.933349:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766346.933351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766346.933354:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766346.933364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766346.933369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766346/real 1775766346] req@ffff9d81984a2d80 x1862023066401152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766346.933376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066401152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766346.933381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766346.933383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766346.933385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766346.933402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766346.933412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766346.933418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766346/real 1775766346] req@ffff9d81984a0700 x1862023066401280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766346.933430:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066401280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766346.933436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766346.933438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766346.933440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766346.933443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766346.933453:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766346.933458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766346/real 1775766346] req@ffff9d81984a3b80 x1862023066401408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766346.933465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066401408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766346.933471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766346.933472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766346.933474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766347.573058:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766347.956952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766348.597065:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766348.981034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766349.621081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766350.004940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766350.645037:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766351.029006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766351.669071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766352.052967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766352.053026:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766352.053035:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766352.053040:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766352.053050:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3183 00000100:00080000:1.0:1775766352.053061:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766352.053076:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766352.053122:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066401536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766352.053154:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766352.053158:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766352.053160:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766352.053165:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3183 00000100:00080000:1.0:1775766352.053169:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766352.053174:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766352.053183:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766352.053198:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066401664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766352.053210:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066401536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766352.053217:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766352.053222:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766352.053224:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766352.053230:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3183 00000100:00080000:1.0:1775766352.053238:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766352.053243:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766352.053249:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766352.053266:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066401792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766352.053278:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766352.053281:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766352.053283:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766352.053288:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3183 00000800:00000100:3.0:1775766352.053294:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766352.053296:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766352.053302:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766352.053311:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766352.053325:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066401920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766352.053336:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766352.053339:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766352.053341:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766352.053347:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3183 00000100:00080000:1.0:1775766352.053352:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766352.053354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766352/real 1775766352] req@ffff9d8190551880 x1862023066401536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766352.053363:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766352.053379:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766352.053407:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066401536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766352.053411:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066402048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766352.053441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766352.053448:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766352.053453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066401536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766352.053510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766352.053530:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766352.053550:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766352/real 1775766352] req@ffff9d8190553480 x1862023066401664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766352.053561:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066401664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766352.053570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766352.053574:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766352.053577:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766352.053584:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766352.053605:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766352.053617:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766352/real 1775766352] req@ffff9d8190553100 x1862023066401792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766352.053637:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066401792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766352.053642:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766352.053653:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766352.053677:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766352.053683:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766352.053701:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766352.053709:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766352/real 1775766352] req@ffff9d8190550e00 x1862023066401920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766352.053717:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066401920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766352.053724:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766352.053726:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766352.053728:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066401920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766352.053732:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766352.053743:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766352.053748:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766352/real 1775766352] req@ffff9d8190551500 x1862023066402048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766352.053756:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066402048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766352.053761:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766352.053763:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766352.053765:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766352.693063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766353.076991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766353.717046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766354.100957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766354.741034:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766355.125001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766355.765069:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766356.149006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766356.789047:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766357.172962:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766357.172970:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766357.172974:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766357.172978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766357.172983:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3188 00000100:00080000:1.0:1775766357.172993:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766357.172998:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766357.173034:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066402176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766357.173060:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766357.173063:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766357.173064:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766357.173067:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3188 00000100:00080000:1.0:1775766357.173069:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766357.173072:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766357.173079:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766357.173086:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066402304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766357.173093:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766357.173095:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766357.173096:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766357.173099:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3188 00000100:00080000:1.0:1775766357.173101:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766357.173103:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766357.173108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766357.173114:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066402432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766357.173121:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766357.173123:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766357.173124:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766357.173128:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3188 00000100:00080000:1.0:1775766357.173131:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766357.173132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066402176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766357.173134:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766357.173140:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766357.173146:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066402560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766357.173152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766357.173155:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766357.173156:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766357.173158:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3188 00000100:00080000:1.0:1775766357.173161:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766357.173163:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766357.173166:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766357.173173:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066402688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766357.173200:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766357.173247:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766357/real 1775766357] req@ffff9d8190551500 x1862023066402176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766357.173266:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066402176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766357.173278:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766357.173282:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766357.173286:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066402176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766357.173294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766357.173316:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766357.173328:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766357/real 1775766357] req@ffff9d8190550e00 x1862023066402304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766357.173340:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066402304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766357.173349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766357.173352:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766357.173356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766357.173376:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766357.173394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766357.173405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766357/real 1775766357] req@ffff9d8190553100 x1862023066402432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766357.173424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066402432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766357.173432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766357.173435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766357.173438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766357.173443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766357.173460:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766357.173469:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766357/real 1775766357] req@ffff9d8190553480 x1862023066402560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766357.173479:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066402560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766357.173486:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766357.173489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766357.173491:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766357.173496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766357.173511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766357.173521:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766357/real 1775766357] req@ffff9d8190551880 x1862023066402688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766357.173532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066402688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766357.173546:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766357.173550:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766357.173554:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766357.813071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766358.196999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766358.837116:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766359.220954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766359.861031:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766360.244970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766360.885090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766361.268974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766361.909018:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766362.292992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766362.292996:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766362.293004:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766362.293009:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766362.293019:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3193 00000100:00080000:2.0:1775766362.293030:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766362.293038:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766362.293078:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066402816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766362.293108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766362.293112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766362.293114:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766362.293119:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3193 00000100:00080000:2.0:1775766362.293124:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766362.293129:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766362.293138:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766362.293141:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066402816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766362.293157:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066402944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766362.293170:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766362.293174:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766362.293176:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766362.293180:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3193 00000100:00080000:2.0:1775766362.293186:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766362.293190:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766362.293196:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766362.293203:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766362.293211:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066403072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766362.293223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766362.293227:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766362.293229:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766362.293234:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3193 00000100:00080000:2.0:1775766362.293238:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766362.293243:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766362.293249:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766362.293257:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766362/real 1775766362] req@ffff9d81984a2a00 x1862023066402816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766362.293262:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066403200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766362.293280:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766362.293285:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066402816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766362.293286:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766362.293290:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766362.293300:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3193 00000100:00080000:3.0:1775766362.293304:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766362.293305:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766362.293312:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775766362.293313:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766362.293319:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066402816:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775766362.293321:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766362.293336:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066403328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766362.293341:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766362.293369:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766362.293381:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766362/real 1775766362] req@ffff9d81984a3b80 x1862023066402944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766362.293390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066402944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766362.293396:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766362.293398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766362.293401:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066402944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766362.293405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766362.293415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766362.293421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766362/real 1775766362] req@ffff9d81984a0700 x1862023066403072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766362.293429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066403072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766362.293434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766362.293436:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766362.293438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766362.293441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766362.293451:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766362.293457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766362/real 1775766362] req@ffff9d81984a2d80 x1862023066403200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766362.293464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066403200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766362.293470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766362.293472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766362.293474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766362.293483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766362.293521:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766362.293530:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766362/real 1775766362] req@ffff9d81984a3480 x1862023066403328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766362.293539:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066403328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766362.293545:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766362.293547:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766362.293549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766362.933046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766363.316963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766363.957021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766364.340985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766364.981085:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766365.364968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766366.005065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766366.388970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766367.029104:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766367.413023:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766367.413034:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766367.413040:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766367.413064:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3198 00000100:00080000:1.0:1775766367.413078:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766367.413084:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766367.413091:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766367.413197:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066403456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766367.413235:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766367.413249:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766367.413252:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766367.413256:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3198 00000100:00080000:1.0:1775766367.413261:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766367.413266:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766367.413277:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766367.413293:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066403584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766367.413320:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766367.413324:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766367.413328:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766367.413332:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066403456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766367.413338:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3198 00000100:00080000:1.0:1775766367.413348:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766367.413352:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766367.413357:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766367.413373:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066403712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766367.413385:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766367.413392:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766367.413394:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766367.413400:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3198 00000100:00080000:1.0:1775766367.413407:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766367.413413:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775766367.413418:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775766367.413429:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766367.413457:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066403840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766367.413475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766367/real 1775766367] req@ffff9d8190551880 x1862023066403456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766367.413477:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766367.413492:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766367.413498:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766367.413536:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3198 00000100:00100000:3.0:1775766367.413545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066403456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766367.413546:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766367.413563:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766367.413577:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775766367.413579:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766367.413589:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766367.413594:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066403456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766367.413601:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066403968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766367.413629:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766367.413660:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766367.413672:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766367/real 1775766367] req@ffff9d8190553480 x1862023066403584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766367.413687:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066403584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766367.413696:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766367.413700:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766367.413703:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766367.413709:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766367.413733:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766367.413742:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766367/real 1775766367] req@ffff9d8190553100 x1862023066403712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766367.413754:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066403712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766367.413762:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766367.413765:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766367.413768:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766367.413774:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766367.413790:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766367.413799:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766367/real 1775766367] req@ffff9d8190550e00 x1862023066403840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766367.413810:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066403840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766367.413827:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766367.413831:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766367.413834:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766367.413839:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766367.413854:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766367.413862:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766367/real 1775766367] req@ffff9d8190551500 x1862023066403968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766367.413873:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066403968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766367.413914:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766367.413918:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766367.413921:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066403968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766368.053079:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766368.437038:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766369.077185:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766369.460941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766370.101051:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766370.484995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766371.125147:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766371.509018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766372.149075:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766372.533004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766372.533049:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766372.533058:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766372.533062:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766372.533069:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3204 00000100:00080000:3.0:1775766372.533078:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766372.533083:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766372.533117:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066404096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766372.533152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766372.533157:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766372.533159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766372.533163:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3204 00000100:00080000:3.0:1775766372.533169:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766372.533173:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766372.533182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766372.533195:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066404224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766372.533206:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766372.533210:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766372.533212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766372.533215:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3204 00000100:00080000:3.0:1775766372.533218:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766372.533221:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766372.533226:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766372.533237:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066404352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766372.533238:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066404096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766372.533262:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766372.533264:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766372.533266:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766372.533268:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3204 00000100:00080000:3.0:1775766372.533272:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766372.533274:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766372.533279:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766372.533287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066404480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766372.533293:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766372.533296:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766372.533297:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766372.533302:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766372.533302:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3204 00000100:00080000:3.0:1775766372.533305:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766372.533308:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766372.533311:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766372.533319:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066404608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766372.533352:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766372/real 1775766372] req@ffff9d8188740e00 x1862023066404096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766372.533370:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066404096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766372.533378:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766372.533382:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766372.533385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066404096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766372.533391:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766372.533405:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766372.533412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766372/real 1775766372] req@ffff9d8188743100 x1862023066404224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766372.533420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066404224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766372.533425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766372.533427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766372.533428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766372.533445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766372.533478:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766372.533489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766372/real 1775766372] req@ffff9d8188741180 x1862023066404352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766372.533509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066404352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766372.533518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766372.533521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766372.533525:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766372.533531:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766372.533549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766372.533557:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766372/real 1775766372] req@ffff9d8188741880 x1862023066404480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766372.533565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066404480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766372.533571:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766372.533573:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766372.533575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766372.533578:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766372.533588:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766372.533593:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766372/real 1775766372] req@ffff9d8188742680 x1862023066404608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766372.533600:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066404608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766372.533613:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766372.533617:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766372.533620:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766373.173122:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766373.556977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766374.197077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766374.580979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766375.221042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766375.604958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766376.245032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766376.628981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766377.269163:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766377.652998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766377.653027:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766377.653037:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766377.653044:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766377.653056:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3209 00000100:00080000:3.0:1775766377.653069:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766377.653078:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766377.653125:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066404736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766377.653164:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766377.653169:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766377.653172:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766377.653177:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3209 00000100:00080000:3.0:1775766377.653181:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766377.653186:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766377.653199:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766377.653216:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066404864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766377.653225:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066404736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766377.653246:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766377.653250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766377.653252:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766377.653260:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3209 00000100:00080000:3.0:1775766377.653265:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766377.653268:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766377.653276:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766377.653293:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066404992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766377.653305:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766377.653310:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766377.653312:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766377.653317:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3209 00000100:00080000:3.0:1775766377.653323:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766377.653324:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766377.653335:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766377.653342:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766377.653359:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066405120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766377.653371:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766377.653376:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766377.653379:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766377.653385:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3209 00000100:00080000:3.0:1775766377.653390:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766377.653398:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766377.653399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766377/real 1775766377] req@ffff9d8188743800 x1862023066404736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775766377.653404:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766377.653462:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066405248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766377.653467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066404736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766377.653527:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766377.653537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766377.653544:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066404736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766377.653575:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766377.653606:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766377.653621:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766377/real 1775766377] req@ffff9d8188741f80 x1862023066404864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766377.653636:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066404864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766377.653647:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766377.653652:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766377.653656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766377.653663:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766377.653681:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766377.653702:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766377/real 1775766377] req@ffff9d8188742300 x1862023066404992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766377.653716:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066404992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766377.653726:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766377.653730:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766377.653733:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066404992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766377.653740:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766377.653759:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766377.653769:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766377/real 1775766377] req@ffff9d8188741c00 x1862023066405120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766377.653782:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066405120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766377.653793:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766377.653796:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766377.653801:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766377.653808:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766377.653826:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766377.653835:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766377/real 1775766377] req@ffff9d8188742a00 x1862023066405248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766377.653848:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066405248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766377.653858:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766377.653861:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766377.653865:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766378.293178:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766378.677048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766379.317082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766379.701008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766380.341125:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766380.724980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766381.365027:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766381.748966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766382.389159:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766382.772963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766382.772996:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766382.773003:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766382.773008:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766382.773018:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3214 00000100:00080000:2.0:1775766382.773029:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766382.773037:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766382.773081:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066405376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766382.773111:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766382.773115:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766382.773118:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766382.773122:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3214 00000100:00080000:2.0:1775766382.773128:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766382.773133:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766382.773144:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766382.773148:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066405376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766382.773158:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066405504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766382.773169:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766382.773173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766382.773175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766382.773180:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3214 00000100:00080000:2.0:1775766382.773198:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766382.773202:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775766382.773208:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775766382.773208:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766382.773218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066405632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766382.773229:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766382.773232:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766382.773234:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766382.773238:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3214 00000100:00080000:2.0:1775766382.773243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766382.773246:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766382.773252:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766382.773262:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066405760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766382.773267:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766382/real 1775766382] req@ffff9d81984a2300 x1862023066405376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766382.773273:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766382.773277:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766382.773279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766382.773283:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3214 00000100:00100000:1.0:1775766382.773288:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066405376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766382.773293:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766382.773297:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:1.0:1775766382.773299:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775766382.773302:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:1.0:1775766382.773304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766382.773310:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066405376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766382.773313:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066405888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766382.773336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766382.773383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766382.773397:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766382/real 1775766382] req@ffff9d81984a1f80 x1862023066405504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766382.773407:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066405504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766382.773414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766382.773416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766382.773419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766382.773424:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766382.773440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766382.773447:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766382/real 1775766382] req@ffff9d81984a1180 x1862023066405632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766382.773456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066405632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766382.773462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766382.773464:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766382.773466:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766382.773470:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766382.773482:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766382.773489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766382/real 1775766382] req@ffff9d81984a0000 x1862023066405760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766382.773497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066405760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766382.773504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766382.773506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766382.773508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766382.773512:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766382.773523:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766382.773529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766382/real 1775766382] req@ffff9d81984a2680 x1862023066405888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766382.773538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066405888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766382.773544:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766382.773546:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766382.773548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066405888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766383.413016:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766383.796963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766384.437118:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766384.821012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766385.461156:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766385.845028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766386.485066:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766386.868963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766387.509030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766387.893025:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766387.893036:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766387.893041:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766387.893055:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766387.893055:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3219 00000100:00080000:2.0:1775766387.893070:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766387.893080:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766387.893123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066406016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766387.893167:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766387.893171:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766387.893173:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766387.893177:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3219 00000100:00080000:2.0:1775766387.893180:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766387.893183:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766387.893193:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766387.893206:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066406144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766387.893218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766387.893222:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766387.893225:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766387.893234:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3219 00000100:00100000:1.0:1775766387.893235:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066406016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766387.893240:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766387.893245:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766387.893260:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766387.893280:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066406272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766387.893290:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766387.893294:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766387.893295:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766387.893299:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3219 00000100:00080000:2.0:1775766387.893302:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766387.893305:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766387.893310:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766387.893345:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066406400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766387.893353:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766387.893356:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766387.893358:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766387.893363:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3219 00000100:00080000:2.0:1775766387.893366:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766387.893369:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766387.893373:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766387.893385:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066406528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766387.893386:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766387.893478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766387/real 1775766387] req@ffff9d81984a0e00 x1862023066406016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766387.893507:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066406016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766387.893520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766387.893525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766387.893529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066406016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766387.893574:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766387.893601:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766387.893615:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766387/real 1775766387] req@ffff9d81984a1880 x1862023066406144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766387.893626:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066406144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766387.893633:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766387.893636:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766387.893639:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766387.893644:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766387.893659:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766387.893666:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766387/real 1775766387] req@ffff9d81984a3800 x1862023066406272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766387.893676:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066406272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766387.893689:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766387.893692:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766387.893694:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766387.893699:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766387.893729:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766387.893738:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766387/real 1775766387] req@ffff9d81984a3100 x1862023066406400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766387.893751:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066406400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766387.893762:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766387.893767:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766387.893770:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766387.893795:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766387.893821:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766387.893834:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766387/real 1775766387] req@ffff9d81984a1500 x1862023066406528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766387.893844:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066406528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766387.893852:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766387.893855:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766387.893858:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766388.533008:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766388.917018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766389.557052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766389.940988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766390.581156:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766390.964973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766391.605048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766391.988986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766392.629068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766393.013000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766393.013031:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766393.013046:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766393.013055:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766393.013071:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3224 00000100:00080000:3.0:1775766393.013089:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766393.013099:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766393.013159:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066406656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766393.013202:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766393.013208:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766393.013210:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766393.013215:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3224 00000100:00080000:3.0:1775766393.013220:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766393.013226:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766393.013240:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766393.013260:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066406784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766393.013304:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066406656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766393.013305:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766393.013314:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766393.013320:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766393.013331:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3224 00000100:00080000:3.0:1775766393.013337:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766393.013341:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766393.013348:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766393.013368:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066406912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766393.013381:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766393.013388:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766393.013391:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766393.013398:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3224 00000100:00080000:3.0:1775766393.013417:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766393.013425:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766393.013436:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766393.013463:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066407040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766393.013487:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766393.013493:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766393.013498:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766393.013511:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3224 00000100:00100000:1.0:1775766393.013516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406784:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775766393.013519:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766393.013533:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766393.013543:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766393.013566:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066407168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766393.013587:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766393.013691:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766393.013720:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766393.653072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766394.037072:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766394.677161:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766395.061003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766395.701109:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766396.085012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775766396.085296:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775766396.085361:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775766396.085388:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775766396.085443:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775766396.085456:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775766396.085470:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775766396.085482:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775766396.085518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766393/real 1775766396] req@ffff9d8188741c00 x1862023066407168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766396.085569:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066407168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766396.085591:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766396.085600:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766396.085608:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407168:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766396.085623:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766393/real 1775766396] req@ffff9d8188740380 x1862023066406784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766396.085650:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066406784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766396.085666:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766396.085674:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766396.085680:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766396.085694:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766393/real 1775766396] req@ffff9d8188740000 x1862023066406912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766396.085720:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066406912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766396.085736:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766396.085742:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766396.085749:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066406912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766396.085762:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766393/real 1775766396] req@ffff9d8188742a00 x1862023066407040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766396.085787:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066407040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766396.085805:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766396.085811:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766396.085818:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766396.085831:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766393/real 1775766396] req@ffff9d8188740700 x1862023066406656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766396.085856:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066406656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766396.085872:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766396.085920:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766396.085928:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066406656:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775766396.725031:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766397.109048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766397.749101:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766398.133026:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766398.133039:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766398.133045:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766398.133057:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3229 00000100:00080000:3.0:1775766398.133072:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766398.133081:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766398.133114:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775766398.133132:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066407296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766398.133174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766398.133179:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766398.133182:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766398.133187:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3229 00000100:00080000:3.0:1775766398.133191:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766398.133196:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766398.133209:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766398.133263:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066407424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766398.133283:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766398.133289:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766398.133294:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766398.133303:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3229 00000100:00080000:3.0:1775766398.133311:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766398.133320:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766398.133331:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766398.133360:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066407552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766398.133362:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066407296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766398.133400:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766398.133407:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766398.133411:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766398.133421:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3229 00000100:00080000:3.0:1775766398.133430:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766398.133437:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766398.133449:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775766398.133480:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766398.133487:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066407680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766398.133511:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766398.133518:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766398.133531:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766398.133540:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3229 00000100:00080000:3.0:1775766398.133548:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766398.133556:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766398.133568:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766398.133583:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766398/real 1775766398] req@ffff9d8188742300 x1862023066407296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766398.133595:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066407808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766398.133654:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066407296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766398.133677:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766398.133687:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766398.133696:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066407296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766398.133712:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766398.133753:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766398.133772:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766398/real 1775766398] req@ffff9d8188741f80 x1862023066407424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766398.133790:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066407424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766398.133820:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766398.133828:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766398.133834:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766398.133931:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766398.133976:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766398.133998:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766398/real 1775766398] req@ffff9d8188743800 x1862023066407552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766398.134022:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066407552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766398.134038:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766398.134044:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766398.134050:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766398.134061:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766398.134098:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766398.134117:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766398/real 1775766398] req@ffff9d8188742680 x1862023066407680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766398.134157:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066407680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766398.134174:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766398.134181:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766398.134188:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766398.134198:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766398.134263:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766398.134285:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766398/real 1775766398] req@ffff9d8188741880 x1862023066407808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766398.134308:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066407808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766398.134323:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766398.134329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766398.134334:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066407808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766398.773108:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766399.157076:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766399.797088:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766400.180991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766400.821142:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766401.205003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766401.845051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766402.228940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766402.869101:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766403.252957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766403.252971:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766403.252979:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766403.252984:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766403.252994:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3234 00000100:00080000:3.0:1775766403.253006:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766403.253013:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766403.253056:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066407936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766403.253104:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766403.253109:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766403.253111:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766403.253115:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3234 00000100:00080000:3.0:1775766403.253120:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766403.253125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766403.253135:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766403.253150:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066408064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766403.253177:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766403.253181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766403.253183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766403.253188:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066407936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766403.253193:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3234 00000100:00080000:3.0:1775766403.253199:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766403.253204:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766403.253220:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766403.253236:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066408192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766403.253249:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766403.253254:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766403.253256:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766403.253263:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3234 00000800:00000100:1.0:1775766403.253266:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766403.253271:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766403.253277:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766403.253284:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766403.253300:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066408320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766403.253319:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766403.253323:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766403.253325:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766403.253329:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3234 00000100:00080000:3.0:1775766403.253334:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766403.253339:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766403.253345:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766403.253346:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766403/real 1775766403] req@ffff9d8188741180 x1862023066407936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766403.253364:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066408448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766403.253370:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066407936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766403.253394:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766403.253399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766403.253403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066407936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766403.253410:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766403.253431:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766403.253441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766403/real 1775766403] req@ffff9d8188743100 x1862023066408064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766403.253450:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066408064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766403.253456:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766403.253459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766403.253461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766403.253479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766403.253491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766403.253497:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766403/real 1775766403] req@ffff9d8188740e00 x1862023066408192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766403.253505:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066408192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766403.253511:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766403.253513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766403.253515:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766403.253518:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766403.253529:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766403.253535:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766403/real 1775766403] req@ffff9d8188740a80 x1862023066408320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766403.253553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066408320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766403.253560:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766403.253562:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766403.253564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766403.253568:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766403.253578:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766403.253583:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766403/real 1775766403] req@ffff9d8188741500 x1862023066408448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766403.253591:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066408448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766403.253597:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766403.253599:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766403.253600:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766403.893134:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766404.277046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766404.917089:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766405.301025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766405.941056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766406.324941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766406.965208:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766407.348972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766407.989071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766408.372942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766408.372957:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766408.372962:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766408.372967:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766408.372974:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3239 00000100:00080000:3.0:1775766408.372983:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766408.372988:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766408.373021:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066408576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766408.373047:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766408.373050:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766408.373051:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766408.373053:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3239 00000100:00080000:3.0:1775766408.373056:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766408.373064:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766408.373072:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766408.373080:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066408704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766408.373087:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766408.373089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766408.373090:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766408.373093:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3239 00000100:00080000:3.0:1775766408.373095:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766408.373097:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766408.373100:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766408.373122:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066408832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766408.373132:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766408.373133:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066408576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766408.373135:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766408.373138:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766408.373145:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3239 00000100:00080000:3.0:1775766408.373147:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766408.373150:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766408.373153:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766408.373165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066408960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766408.373177:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766408.373179:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766408.373180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766408.373182:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3239 00000100:00080000:3.0:1775766408.373190:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766408.373192:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766408.373196:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:1.0:1775766408.373200:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766408.373206:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066409088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766408.373246:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766408/real 1775766408] req@ffff9d8188743480 x1862023066408576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766408.373265:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066408576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766408.373275:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766408.373280:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766408.373284:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066408576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766408.373292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766408.373314:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766408.373325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766408/real 1775766408] req@ffff9d8188742d80 x1862023066408704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766408.373338:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066408704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766408.373346:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766408.373350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766408.373353:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766408.373376:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766408.373396:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766408.373407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766408/real 1775766408] req@ffff9d8188743b80 x1862023066408832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766408.373419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066408832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766408.373428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766408.373431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766408.373435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766408.373441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766408.373458:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766408.373468:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766408/real 1775766408] req@ffff9d8188741500 x1862023066408960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766408.373480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066408960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766408.373490:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766408.373493:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766408.373496:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066408960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766408.373502:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766408.373518:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766408.373527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766408/real 1775766408] req@ffff9d8188740a80 x1862023066409088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766408.373538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066409088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766408.373547:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766408.373550:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766408.373553:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766409.013024:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766409.396971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766410.037190:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766410.420989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766411.061145:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766411.445036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766412.085115:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766412.468991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766413.109192:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766413.493009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766413.493172:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766413.493180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766413.493185:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766413.493194:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3245 00000100:00080000:2.0:1775766413.493206:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766413.493215:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766413.493258:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066409216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766413.493284:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766413.493288:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766413.493289:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766413.493293:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3245 00000100:00080000:2.0:1775766413.493296:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766413.493301:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766413.493311:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766413.493327:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066409216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766413.493328:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066409344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766413.493347:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766413.493350:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766413.493352:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766413.493356:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3245 00000100:00080000:2.0:1775766413.493359:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766413.493362:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766413.493366:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766413.493379:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066409472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766413.493387:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766413.493391:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766413.493393:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766413.493396:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3245 00000100:00080000:2.0:1775766413.493400:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766413.493406:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766413.493408:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766413.493415:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766413.493426:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066409600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766413.493436:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766413.493439:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766413.493441:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766413.493445:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3245 00000100:00080000:2.0:1775766413.493450:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766413.493454:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766413.493459:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766413.493465:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766413/real 1775766413] req@ffff9d81984a0a80 x1862023066409216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766413.493475:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066409728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766413.493499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066409216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766413.493512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766413.493534:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766413.493539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066409216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766413.493565:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766413.493592:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766413.493605:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766413/real 1775766413] req@ffff9d81984a1500 x1862023066409344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766413.493618:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066409344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766413.493627:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766413.493630:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766413.493634:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766413.493640:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766413.493656:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766413.493676:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766413/real 1775766413] req@ffff9d81984a3100 x1862023066409472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766413.493688:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066409472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766413.493697:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766413.493700:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766413.493704:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766413.493710:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766413.493730:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766413.493746:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766413/real 1775766413] req@ffff9d81984a3800 x1862023066409600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766413.493766:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066409600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766413.493781:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766413.493786:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766413.493792:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766413.493802:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766413.493844:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766413.493860:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766413/real 1775766413] req@ffff9d81984a1880 x1862023066409728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766413.493873:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066409728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766413.493882:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766413.493885:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766413.493926:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766414.133129:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766414.516992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766415.157003:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766415.540967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766416.181001:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766416.564953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766417.205119:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766417.588970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766418.229155:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766418.613146:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766418.613146:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766418.613160:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766418.613165:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766418.613179:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3250 00000100:00080000:3.0:1775766418.613212:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766418.613221:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766418.613269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066409856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766418.613325:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766418.613330:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766418.613332:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766418.613339:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3250 00000100:00080000:3.0:1775766418.613345:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766418.613349:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766418.613358:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766418.613372:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066409984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766418.613381:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766418.613384:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766418.613386:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766418.613390:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3250 00000100:00080000:3.0:1775766418.613394:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766418.613397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066409856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766418.613400:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766418.613409:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766418.613420:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066410112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766418.613428:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766418.613432:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766418.613434:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766418.613437:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3250 00000100:00080000:3.0:1775766418.613468:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766418.613471:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766418.613476:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775766418.613480:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766418.613493:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066410240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766418.613502:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766418.613505:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766418.613506:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766418.613510:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3250 00000100:00080000:3.0:1775766418.613514:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766418.613530:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766418.613534:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766418.613544:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766418/real 1775766418] req@ffff9d8188740e00 x1862023066409856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766418.613547:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066410368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766418.613610:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066409856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766418.613621:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766418.613625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766418.613630:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066409856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766418.613637:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766418.613658:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766418.613669:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766418/real 1775766418] req@ffff9d8188743100 x1862023066409984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766418.613680:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066409984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766418.613687:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766418.613690:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766418.613693:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066409984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766418.613714:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766418.613728:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766418.613736:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766418/real 1775766418] req@ffff9d8188741180 x1862023066410112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766418.613746:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066410112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766418.613759:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766418.613762:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766418.613765:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766418.613770:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766418.613785:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766418.613797:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766418/real 1775766418] req@ffff9d8188741880 x1862023066410240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766418.613813:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066410240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766418.613826:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766418.613831:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766418.613835:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766418.613844:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766418.613889:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766418.613955:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766418/real 1775766418] req@ffff9d8188742680 x1862023066410368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766418.613972:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066410368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766418.613983:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766418.613987:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766418.613992:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766419.253141:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766419.636973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766420.277106:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766420.660955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766421.301179:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766421.685017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766422.325108:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766422.708973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766423.349120:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766423.732982:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766423.732993:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766423.733000:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766423.733011:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3255 00000100:00100000:0.0:1775766423.733012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766423.733026:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766423.733036:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766423.733082:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066410496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766423.733116:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766423.733120:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766423.733122:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766423.733126:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3255 00000100:00080000:3.0:1775766423.733130:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766423.733134:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766423.733145:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766423.733159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066410624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766423.733171:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766423.733176:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766423.733178:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766423.733183:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3255 00000100:00080000:3.0:1775766423.733186:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766423.733194:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766423.733197:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066410496:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775766423.733200:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766423.733219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066410752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766423.733232:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766423.733235:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766423.733237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766423.733242:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3255 00000100:00080000:3.0:1775766423.733246:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766423.733262:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766423.733267:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775766423.733291:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766423.733295:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066410880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766423.733316:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766423.733320:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766423.733322:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766423.733326:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3255 00000100:00080000:3.0:1775766423.733331:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766423.733336:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766423.733343:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766423.733357:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766423/real 1775766423] req@ffff9d8188743800 x1862023066410496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766423.733361:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066411008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766423.733426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066410496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766423.733446:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766423.733454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766423.733460:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066410496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766423.733500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766423.733540:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766423.733559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766423/real 1775766423] req@ffff9d8188741f80 x1862023066410624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766423.733582:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066410624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766423.733606:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766423.733612:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766423.733618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766423.733671:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766423.733703:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766423.733724:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766423/real 1775766423] req@ffff9d8188742300 x1862023066410752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766423.733745:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066410752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766423.733760:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766423.733765:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766423.733770:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766423.733781:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766423.733817:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766423.733836:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766423/real 1775766423] req@ffff9d8188740700 x1862023066410880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766423.733857:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066410880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766423.733870:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766423.733875:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766423.733880:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066410880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766423.733889:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766423.733993:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766423.734014:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766423/real 1775766423] req@ffff9d8188742a00 x1862023066411008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766423.734034:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066411008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766423.734049:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766423.734055:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766423.734060:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766424.373042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766424.756971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766425.397129:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766425.780954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766426.421100:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766426.805057:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766427.445111:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766427.829009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766428.469085:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766428.852999:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766428.853000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766428.853009:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766428.853015:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766428.853024:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3260 00000100:00080000:3.0:1775766428.853035:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766428.853043:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766428.853084:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066411136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766428.853115:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766428.853120:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766428.853122:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766428.853126:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3260 00000100:00080000:3.0:1775766428.853131:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766428.853135:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766428.853145:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766428.853158:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066411136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766428.853161:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066411264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766428.853175:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766428.853179:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766428.853181:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766428.853186:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3260 00000100:00080000:3.0:1775766428.853204:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766428.853209:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766428.853215:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775766428.853221:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766428.853234:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066411392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766428.853246:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766428.853250:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766428.853252:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766428.853257:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3260 00000100:00100000:1.0:1775766428.853262:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766428/real 1775766428] req@ffff9d8188740000 x1862023066411136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766428.853263:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766428.853271:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766428.853279:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766428.853284:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066411136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766428.853292:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775766428.853297:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066411520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766428.853298:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766428.853303:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066411136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766428.853324:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766428.853327:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411264:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775766428.853332:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766428.853335:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766428.853340:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3260 00000100:00080000:3.0:1775766428.853347:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766428.853350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766428.853355:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766428.853362:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766428.853363:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766428/real 1775766428] req@ffff9d8188740380 x1862023066411264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766428.853376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066411264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766428.853382:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066411648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766428.853385:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766428.853392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766428.853398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766428.853409:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766428.853424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766428.853432:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766428/real 1775766428] req@ffff9d8188741c00 x1862023066411392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766428.853441:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066411392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766428.853446:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766428.853449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766428.853452:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766428.853455:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766428.853465:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766428.853470:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766428/real 1775766428] req@ffff9d8188742a00 x1862023066411520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766428.853477:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066411520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766428.853483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766428.853485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766428.853486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766428.853495:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766428.853505:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766428.853510:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766428/real 1775766428] req@ffff9d8188740700 x1862023066411648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766428.853518:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066411648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766428.853523:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766428.853525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766428.853527:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766429.493071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766429.876942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766430.517056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766430.901000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766431.541065:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766431.924991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766432.565030:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766432.948974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766433.589076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766433.972998:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766433.973012:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766433.973019:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766433.973024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766433.973032:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3265 00000100:00080000:1.0:1775766433.973045:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766433.973053:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766433.973114:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066411776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766433.973146:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766433.973149:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766433.973151:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766433.973154:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3265 00000100:00080000:1.0:1775766433.973157:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766433.973161:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766433.973171:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766433.973182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066411904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766433.973208:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766433.973208:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066411776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766433.973216:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766433.973219:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766433.973223:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3265 00000100:00080000:1.0:1775766433.973226:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766433.973228:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766433.973231:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766433.973241:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066412032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766433.973247:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766433.973250:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766433.973252:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766433.973255:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3265 00000100:00080000:1.0:1775766433.973288:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766433.973292:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766433.973298:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766433.973302:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766433.973316:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066412160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766433.973327:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766433.973331:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766433.973333:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766433.973339:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3265 00000100:00080000:1.0:1775766433.973345:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766433.973349:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766433.973355:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766433.973359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766433/real 1775766433] req@ffff9d8190551500 x1862023066411776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766433.973368:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066412288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766433.973380:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066411776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766433.973391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766433.973395:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766433.973399:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066411776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766433.973423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766433.973440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766433.973457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766433/real 1775766433] req@ffff9d8190550e00 x1862023066411904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766433.973466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066411904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766433.973473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766433.973475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766433.973477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066411904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766433.973482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766433.973494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766433.973500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766433/real 1775766433] req@ffff9d8190553100 x1862023066412032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766433.973508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066412032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766433.973514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766433.973516:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766433.973518:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766433.973521:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766433.973533:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766433.973539:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766433/real 1775766433] req@ffff9d8190553480 x1862023066412160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766433.973546:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066412160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766433.973554:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766433.973556:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766433.973559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766433.973562:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766433.973572:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766433.973577:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766433/real 1775766433] req@ffff9d8190551880 x1862023066412288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766433.973584:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066412288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766433.973590:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766433.973592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766433.973594:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766434.613025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766434.996986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766435.637055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766436.021013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766436.661024:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766437.045017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766437.685081:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766438.068963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766438.709147:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766439.092969:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766439.092977:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766439.092981:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766439.092986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766439.092989:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3270 00000100:00080000:1.0:1775766439.093002:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766439.093007:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766439.093043:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066412416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766439.093078:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766439.093081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766439.093082:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766439.093085:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3270 00000100:00080000:1.0:1775766439.093088:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766439.093091:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766439.093099:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766439.093113:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066412416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766439.093118:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066412544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766439.093127:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766439.093131:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766439.093133:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766439.093136:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3270 00000100:00080000:1.0:1775766439.093139:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766439.093140:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766439.093145:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766439.093153:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066412672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766439.093159:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766439.093161:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766439.093163:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766439.093165:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3270 00000100:00080000:1.0:1775766439.093167:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766439.093169:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766439.093172:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766439.093177:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766439.093182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066412800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766439.093189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766439.093192:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766439.093193:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766439.093197:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3270 00000100:00080000:1.0:1775766439.093199:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766439.093201:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766439.093205:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766439.093211:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066412928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766439.093231:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766439/real 1775766439] req@ffff9d8190551880 x1862023066412416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766439.093263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066412416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766439.093274:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766439.093278:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766439.093282:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066412416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766439.093305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766439.093331:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766439.093345:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766439/real 1775766439] req@ffff9d8190553480 x1862023066412544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766439.093358:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066412544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766439.093367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766439.093370:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766439.093372:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766439.093377:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766439.093393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766439.093401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766439/real 1775766439] req@ffff9d8190553100 x1862023066412672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766439.093409:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066412672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766439.093414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766439.093416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766439.093418:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766439.093422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766439.093432:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766439.093437:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766439/real 1775766439] req@ffff9d8190550e00 x1862023066412800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766439.093449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066412800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766439.093455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766439.093457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766439.093458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766439.093462:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766439.093473:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766439.093482:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766439/real 1775766439] req@ffff9d8190551500 x1862023066412928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766439.093494:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066412928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766439.093503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766439.093506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766439.093509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066412928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766439.733072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766440.117023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766440.757156:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766441.140977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766441.781023:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766442.164983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766442.805112:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766443.188987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766443.829089:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766444.213019:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766444.213019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766444.213031:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766444.213035:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766444.213044:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3275 00000100:00080000:0.0:1775766444.213080:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766444.213087:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766444.213126:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066413056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766444.213154:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766444.213168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766444.213170:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766444.213175:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3275 00000100:00080000:0.0:1775766444.213179:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766444.213184:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766444.213194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766444.213210:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066413184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766444.213232:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766444.213235:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066413056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766444.213236:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766444.213239:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766444.213243:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3275 00000100:00080000:0.0:1775766444.213247:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766444.213250:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766444.213255:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766444.213265:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066413312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766444.213275:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766444.213279:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766444.213280:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766444.213284:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3275 00000100:00080000:0.0:1775766444.213287:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766444.213291:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766444.213296:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766444.213297:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766444.213306:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066413440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766444.213315:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766444.213318:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766444.213331:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766444.213335:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3275 00000100:00080000:0.0:1775766444.213338:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766444.213342:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766444.213346:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766444.213356:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066413568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766444.213357:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766444/real 1775766444] req@ffff9d81901ad180 x1862023066413056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766444.213378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066413056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766444.213397:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766444.213402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766444.213407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066413056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766444.213434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766444.213460:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766444.213488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766444/real 1775766444] req@ffff9d81901ae680 x1862023066413184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766444.213498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066413184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766444.213505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766444.213508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766444.213510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766444.213515:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766444.213530:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766444.213552:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766444/real 1775766444] req@ffff9d81901aed80 x1862023066413312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766444.213561:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066413312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766444.213567:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766444.213569:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766444.213572:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766444.213577:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766444.213590:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766444.213596:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766444/real 1775766444] req@ffff9d81901afb80 x1862023066413440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766444.213604:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066413440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766444.213611:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766444.213613:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766444.213615:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766444.213619:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766444.213647:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766444.213654:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766444/real 1775766444] req@ffff9d81901af100 x1862023066413568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766444.213662:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066413568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766444.213669:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766444.213671:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766444.213673:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766444.853119:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766445.237013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766445.877072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766446.261044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766446.901073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766447.284957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766447.925042:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766448.309025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766448.949093:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766449.332973:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766449.332982:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766449.332988:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766449.332990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766449.332998:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3280 00000100:00080000:1.0:1775766449.333008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766449.333016:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766449.333054:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066413696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766449.333080:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766449.333083:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766449.333084:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766449.333088:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3280 00000100:00080000:1.0:1775766449.333092:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766449.333098:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766449.333108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766449.333116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066413696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766449.333123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066413824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766449.333137:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766449.333139:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766449.333141:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766449.333144:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3280 00000100:00080000:1.0:1775766449.333163:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766449.333166:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766449.333170:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775766449.333176:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766449.333182:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066413952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766449.333190:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766449.333193:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766449.333194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766449.333197:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3280 00000100:00080000:1.0:1775766449.333199:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766449.333201:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766449.333204:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766449.333211:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066414080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766449.333218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766449.333220:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766449.333222:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766449.333226:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3280 00000100:00100000:3.0:1775766449.333226:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766449/real 1775766449] req@ffff9d8190551500 x1862023066413696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766449.333232:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766449.333239:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766449.333244:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766449.333252:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066413696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766449.333254:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066414208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766449.333277:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766449.333283:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766449.333286:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066413696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766449.333306:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766449.333324:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766449.333333:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766449/real 1775766449] req@ffff9d8190550e00 x1862023066413824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766449.333341:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066413824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766449.333348:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766449.333351:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766449.333355:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766449.333361:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766449.333380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766449.333391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766449/real 1775766449] req@ffff9d8190553100 x1862023066413952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766449.333412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066413952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766449.333418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766449.333420:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766449.333422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066413952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766449.333426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766449.333442:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766449.333448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766449/real 1775766449] req@ffff9d8190553480 x1862023066414080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766449.333456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066414080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766449.333478:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766449.333482:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766449.333486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766449.333493:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766449.333514:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766449.333527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766449/real 1775766449] req@ffff9d8190551880 x1862023066414208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766449.333543:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066414208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766449.333558:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766449.333561:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766449.333563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766449.973039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766450.356994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766450.997068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766451.380985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766452.021024:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766452.404984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766453.045004:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766453.428982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766454.069152:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766454.453058:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766454.453069:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766454.453080:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766454.453087:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766454.453103:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3285 00000100:00080000:1.0:1775766454.453119:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766454.453129:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766454.453188:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066414336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766454.453255:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766454.453260:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766454.453262:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766454.453267:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3285 00000100:00080000:1.0:1775766454.453271:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766454.453276:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766454.453291:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766454.453307:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066414464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766454.453322:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766454.453331:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066414336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766454.453332:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766454.453336:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766454.453365:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3285 00000100:00080000:1.0:1775766454.453369:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766454.453373:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766454.453379:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766454.453393:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066414592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766454.453404:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766454.453408:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766454.453410:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766454.453413:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3285 00000100:00080000:1.0:1775766454.453440:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766454.453443:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766454.453448:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766454.453455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766454.453463:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066414720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766454.453473:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766454.453478:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766454.453480:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766454.453485:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3285 00000100:00080000:1.0:1775766454.453489:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766454.453492:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766454.453498:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766454.453517:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066414848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766454.453549:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766454/real 1775766454] req@ffff9d8190551880 x1862023066414336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766454.453574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066414336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766454.453587:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766454.453593:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766454.453598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066414336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766454.453702:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766454.453731:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766454.453746:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766454/real 1775766454] req@ffff9d8190553480 x1862023066414464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766454.453759:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066414464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766454.453768:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766454.453772:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766454.453775:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766454.453781:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766454.453797:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766454.453805:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766454/real 1775766454] req@ffff9d8190553100 x1862023066414592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766454.453834:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066414592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766454.453843:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766454.453846:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766454.453849:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766454.453855:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766454.453875:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766454.453884:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766454/real 1775766454] req@ffff9d8190550e00 x1862023066414720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766454.453969:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066414720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766454.453989:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766454.453996:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766454.454017:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766454.454026:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766454.454075:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766454.454092:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766454/real 1775766454] req@ffff9d8190551500 x1862023066414848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766454.454111:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066414848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766454.454126:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766454.454131:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766454.454136:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066414848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766455.093098:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766455.477008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766456.117171:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766456.500986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766457.141028:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766457.524965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766458.165083:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766458.548970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766459.189072:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766459.573024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766459.573038:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766459.573046:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766459.573051:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766459.573063:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3291 00000100:00080000:2.0:1775766459.573076:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766459.573085:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766459.573135:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066414976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766459.573162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766459.573165:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766459.573167:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766459.573170:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3291 00000100:00080000:2.0:1775766459.573172:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766459.573175:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766459.573186:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766459.573194:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066415104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766459.573202:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766459.573204:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766459.573206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766459.573208:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3291 00000100:00080000:2.0:1775766459.573210:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766459.573212:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766459.573215:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766459.573221:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066415232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766459.573228:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766459.573231:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766459.573233:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766459.573236:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3291 00000100:00080000:2.0:1775766459.573244:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766459.573247:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766459.573251:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766459.573259:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066415360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766459.573268:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766459.573271:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766459.573273:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766459.573285:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3291 00000100:00100000:3.0:1775766459.573289:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066414976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766459.573290:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766459.573304:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766459.573312:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766459.573321:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066415488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766459.573475:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766459.573522:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766459.573566:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766459.573630:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766460.213140:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766460.597149:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766461.237039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766461.621130:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766462.261128:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766462.645147:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775766462.645291:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775766462.645370:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775766462.645403:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775766462.645477:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775766462.645489:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775766462.645501:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775766462.645514:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775766462.645556:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766459/real 1775766462] req@ffff9d81984a1180 x1862023066415360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766462.645593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066415360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766462.645609:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766462.645615:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766462.645621:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766462.645632:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766459/real 1775766462] req@ffff9d81984a1f80 x1862023066415488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766462.645647:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066415488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766462.645657:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766462.645661:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766462.645665:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415488:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766462.645674:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766459/real 1775766462] req@ffff9d81984a0e00 x1862023066414976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766462.645690:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066414976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766462.645699:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766462.645703:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766462.645708:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066414976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766462.645715:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766459/real 1775766462] req@ffff9d81984a2680 x1862023066415104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766462.645730:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066415104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766462.645740:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766462.645744:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766462.645747:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766462.645755:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766459/real 1775766462] req@ffff9d81984a0000 x1862023066415232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766462.645768:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066415232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766462.645776:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766462.645780:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766462.645783:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415232:192.168.202.105@tcp:38:kworker.0 00010000:00010000:0.0:1775766463.285054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766463.669035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766464.309167:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766464.692999:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766464.693010:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766464.693016:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766464.693017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766464.693030:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3296 00000100:00080000:3.0:1775766464.693043:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766464.693051:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766464.693114:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066415616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766464.693148:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766464.693152:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766464.693154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766464.693159:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3296 00000100:00080000:3.0:1775766464.693163:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766464.693167:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766464.693181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766464.693209:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066415616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766464.693212:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066415744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766464.693247:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766464.693252:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766464.693255:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766464.693262:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3296 00000100:00080000:3.0:1775766464.693268:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766464.693274:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766464.693280:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766464.693296:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066415872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766464.693309:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766464.693312:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766464.693318:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766464.693320:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766464.693327:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3296 00000100:00080000:3.0:1775766464.693332:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766464.693338:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766464.693344:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766464.693362:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066416000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766464.693373:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766464.693374:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766464/real 1775766464] req@ffff9d8188742300 x1862023066415616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766464.693394:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766464.693397:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766464.693405:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3296 00000100:00100000:0.0:1775766464.693412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066415616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766464.693415:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766464.693431:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766464.693444:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775766464.693447:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766464.693455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766464.693466:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066416128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766464.693467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066415616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766464.693552:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766464.693585:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766464.693599:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766464/real 1775766464] req@ffff9d8188741f80 x1862023066415744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766464.693613:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066415744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766464.693622:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766464.693625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766464.693629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766464.693636:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766464.693655:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766464.693679:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766464/real 1775766464] req@ffff9d8188743800 x1862023066415872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766464.693697:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066415872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766464.693705:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766464.693708:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766464.693711:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066415872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766464.693718:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766464.693746:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766464.693759:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766464/real 1775766464] req@ffff9d8188742680 x1862023066416000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766464.693772:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066416000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766464.693782:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766464.693786:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766464.693789:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766464.693796:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766464.693816:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766464.693827:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766464/real 1775766464] req@ffff9d8188741880 x1862023066416128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766464.693838:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066416128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766464.693847:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766464.693851:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766464.693854:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766465.333075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766465.717156:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766466.357126:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766466.740957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766467.381084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766467.764982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766468.405131:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766468.788955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766469.429101:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766469.812978:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766469.812985:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775766469.812986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766469.812989:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766469.812999:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3301 00000100:00080000:3.0:1775766469.813008:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766469.813014:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766469.813056:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066416256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766469.813079:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766469.813081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766469.813083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766469.813085:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3301 00000100:00080000:3.0:1775766469.813088:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766469.813090:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766469.813099:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766469.813112:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066416384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766469.813123:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766469.813127:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766469.813130:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766469.813136:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066416256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766469.813138:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3301 00000100:00080000:3.0:1775766469.813146:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766469.813150:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766469.813156:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766469.813166:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066416512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766469.813175:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766469.813178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766469.813179:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766469.813182:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3301 00000100:00080000:3.0:1775766469.813186:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766469.813189:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766469.813192:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766469.813196:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766469.813203:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066416640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766469.813210:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766469.813213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766469.813214:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766469.813216:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3301 00000100:00080000:3.0:1775766469.813218:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766469.813220:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766469.813224:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766469.813233:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066416768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766469.813241:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766469/real 1775766469] req@ffff9d8188741180 x1862023066416256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766469.813257:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066416256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766469.813265:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766469.813268:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766469.813272:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066416256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766469.813277:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766469.813291:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766469.813299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766469/real 1775766469] req@ffff9d8188743100 x1862023066416384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766469.813306:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066416384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766469.813312:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766469.813314:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766469.813316:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766469.813332:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766469.813342:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766469.813347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766469/real 1775766469] req@ffff9d8188740e00 x1862023066416512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766469.813354:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066416512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766469.813359:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766469.813361:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766469.813363:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766469.813366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766469.813376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766469.813382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766469/real 1775766469] req@ffff9d8188740a80 x1862023066416640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766469.813389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066416640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766469.813395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766469.813397:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766469.813398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766469.813402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766469.813411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766469.813416:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766469/real 1775766469] req@ffff9d8188741500 x1862023066416768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766469.813423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066416768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766469.813428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766469.813430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766469.813431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066416768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766470.453231:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766470.837023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766471.477069:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766471.861018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766472.501074:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766472.884965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766473.525114:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766473.908989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766474.549096:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766474.933000:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766474.933011:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766474.933017:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766474.933029:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3306 00000100:00080000:1.0:1775766474.933043:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766474.933066:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766474.933120:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066416896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766474.933147:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766474.933151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766474.933153:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766474.933173:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3306 00000100:00080000:1.0:1775766474.933176:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766474.933180:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766474.933188:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766474.933200:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066417024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766474.933210:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766474.933214:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766474.933216:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766474.933222:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066416896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766474.933223:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3306 00000100:00080000:1.0:1775766474.933230:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766474.933234:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766474.933238:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766474.933249:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066417152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766474.933257:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766474.933260:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766474.933262:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766474.933266:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3306 00000100:00080000:1.0:1775766474.933288:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766474.933293:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766474.933299:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766474.933300:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766474.933320:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066417280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766474.933334:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766474.933339:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766474.933342:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766474.933350:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3306 00000100:00100000:0.0:1775766474.933351:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766474/real 1775766474] req@ffff9d8190551500 x1862023066416896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766474.933358:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766474.933365:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766474.933374:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766474.933388:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066417408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766474.933393:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066416896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766474.933442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775766474.933445:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766474.933449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766474.933455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066416896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766474.933480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766474.933502:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766474.933514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766474/real 1775766474] req@ffff9d8190550e00 x1862023066417024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766474.933525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066417024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766474.933532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766474.933534:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766474.933537:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766474.933542:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766474.933555:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766474.933562:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766474/real 1775766474] req@ffff9d8190553100 x1862023066417152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766474.933572:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066417152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766474.933584:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766474.933587:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766474.933589:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766474.933593:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766474.933621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766474.933629:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766474/real 1775766474] req@ffff9d8190553480 x1862023066417280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766474.933638:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066417280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766474.933646:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766474.933648:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766474.933651:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766474.933655:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766474.933668:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766474.933675:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766474/real 1775766474] req@ffff9d8190551880 x1862023066417408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766474.933683:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066417408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766474.933690:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766474.933692:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766474.933695:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766475.573007:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766475.957001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766476.597039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766476.980997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766477.621168:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766478.005037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766478.645246:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766479.028972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766479.669137:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766480.052985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766480.053003:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766480.053011:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766480.053016:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766480.053025:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3311 00000100:00080000:3.0:1775766480.053036:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766480.053043:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766480.053083:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066417536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766480.053112:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766480.053116:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766480.053118:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766480.053123:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3311 00000100:00080000:3.0:1775766480.053127:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766480.053131:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766480.053141:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766480.053155:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066417664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766480.053160:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066417536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766480.053187:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766480.053190:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766480.053193:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766480.053198:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3311 00000100:00080000:3.0:1775766480.053203:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766480.053207:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766480.053213:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766480.053226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066417792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766480.053232:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766480.053242:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766480.053246:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766480.053248:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766480.053254:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3311 00000100:00080000:3.0:1775766480.053258:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766480.053262:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766480.053268:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766480.053275:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766480/real 1775766480] req@ffff9d8188743b80 x1862023066417536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766480.053283:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066417920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766480.053293:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066417536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766480.053297:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766480.053307:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766480.053310:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766480.053311:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766480.053318:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775766480.053319:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3311 00000100:00100000:0.0:1775766480.053325:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066417536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766480.053326:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766480.053334:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766480.053339:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766480.053345:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766480.053354:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066418048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766480.053389:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766480.053402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766480/real 1775766480] req@ffff9d8188742d80 x1862023066417664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766480.053412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066417664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766480.053418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766480.053420:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766480.053424:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766480.053428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766480.053439:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766480.053445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766480/real 1775766480] req@ffff9d8188743480 x1862023066417792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766480.053452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066417792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766480.053457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766480.053459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766480.053460:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766480.053464:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766480.053475:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766480.053484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766480/real 1775766480] req@ffff9d8188741500 x1862023066417920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766480.053496:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066417920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766480.053505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766480.053508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766480.053511:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066417920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766480.053527:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766480.053546:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766480.053555:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766480/real 1775766480] req@ffff9d8188740a80 x1862023066418048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766480.053562:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066418048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766480.053568:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766480.053570:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766480.053572:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766480.693080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766481.076984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766481.717172:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766482.101067:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766482.741132:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766483.124985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766483.765162:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766484.148993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766484.789145:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766485.172948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766485.173005:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766485.173011:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766485.173015:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766485.173022:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3316 00000100:00080000:1.0:1775766485.173030:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766485.173035:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766485.173081:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066418176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766485.173105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766485.173108:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766485.173109:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766485.173112:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3316 00000100:00080000:1.0:1775766485.173114:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766485.173116:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766485.173124:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766485.173132:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066418304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766485.173140:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766485.173142:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766485.173144:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766485.173149:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3316 00000100:00100000:0.0:1775766485.173151:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066418176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766485.173151:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766485.173158:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766485.173162:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766485.173171:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066418432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766485.173178:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766485.173180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766485.173182:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766485.173185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3316 00000100:00080000:1.0:1775766485.173200:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766485.173215:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766485.173218:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766485.173222:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766485.173227:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066418560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766485.173235:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766485.173237:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766485.173238:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766485.173241:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3316 00000100:00080000:1.0:1775766485.173243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766485.173245:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766485.173248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766485.173256:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066418688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766485.173266:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766485/real 1775766485] req@ffff9d8190551880 x1862023066418176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766485.173282:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066418176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766485.173290:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766485.173293:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766485.173296:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066418176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766485.173300:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766485.173330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766485.173339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766485/real 1775766485] req@ffff9d8190553480 x1862023066418304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766485.173348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066418304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766485.173353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766485.173355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766485.173358:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766485.173374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766485.173385:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766485.173390:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766485/real 1775766485] req@ffff9d8190553100 x1862023066418432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766485.173398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066418432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766485.173403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766485.173404:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766485.173406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766485.173410:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766485.173420:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766485.173425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766485/real 1775766485] req@ffff9d8190550e00 x1862023066418560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766485.173432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066418560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766485.173438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766485.173440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766485.173443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766485.173446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766485.173454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766485.173460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766485/real 1775766485] req@ffff9d8190551500 x1862023066418688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766485.173466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066418688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766485.173472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766485.173474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766485.173476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766485.813028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766486.197066:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766486.837032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766487.220954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766487.861125:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766488.244950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766488.885069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766489.268954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766489.909050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766490.293023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766490.293025:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766490.293035:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766490.293039:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766490.293048:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3321 00000100:00080000:3.0:1775766490.293059:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766490.293066:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766490.293106:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066418816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766490.293136:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766490.293140:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766490.293141:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766490.293145:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3321 00000100:00080000:3.0:1775766490.293154:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766490.293157:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766490.293166:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766490.293177:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066418944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766490.293187:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766490.293191:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766490.293192:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766490.293196:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3321 00000100:00080000:3.0:1775766490.293199:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766490.293202:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766490.293208:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766490.293217:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066418816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766490.293220:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066419072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766490.293265:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766490.293269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766490.293271:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766490.293275:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3321 00000100:00080000:3.0:1775766490.293279:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766490.293282:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766490.293288:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766490.293308:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066419200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766490.293321:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766490.293326:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766490.293332:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766490.293333:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766490.293344:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3321 00000100:00080000:3.0:1775766490.293351:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766490.293358:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766490.293366:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766490.293383:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066419328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766490.293417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766490/real 1775766490] req@ffff9d8188740e00 x1862023066418816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766490.293445:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066418816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766490.293461:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766490.293467:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766490.293474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066418816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766490.293484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766490.293517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766490.293534:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766490/real 1775766490] req@ffff9d8188743100 x1862023066418944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766490.293566:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066418944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766490.293580:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766490.293585:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766490.293590:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066418944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766490.293620:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766490.293645:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766490.293656:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766490/real 1775766490] req@ffff9d8188741180 x1862023066419072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766490.293668:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066419072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766490.293675:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766490.293678:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766490.293682:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766490.293687:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766490.293702:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766490.293710:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766490/real 1775766490] req@ffff9d8188741880 x1862023066419200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766490.293721:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066419200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766490.293729:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766490.293731:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766490.293734:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766490.293738:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766490.293751:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766490.293759:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766490/real 1775766490] req@ffff9d8188742680 x1862023066419328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766490.293768:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066419328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766490.293776:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766490.293778:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766490.293781:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766490.933078:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766491.316963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766491.957100:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766492.341005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766492.981056:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766493.364979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766494.005160:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766494.388993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766495.029041:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766495.412937:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766495.412952:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766495.412959:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766495.412964:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766495.412975:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3326 00000100:00080000:3.0:1775766495.412985:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766495.412993:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766495.413034:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066419456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766495.413064:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766495.413068:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766495.413070:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766495.413074:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3326 00000100:00080000:3.0:1775766495.413079:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766495.413083:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766495.413093:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766495.413105:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066419456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766495.413110:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066419584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766495.413123:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766495.413127:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766495.413129:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766495.413134:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3326 00000100:00080000:3.0:1775766495.413153:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766495.413158:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766495.413163:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766495.413166:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766495.413183:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066419712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766495.413216:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766495.413220:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766495.413223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766495.413226:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766495/real 1775766495] req@ffff9d8188743800 x1862023066419456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766495.413230:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3326 00000100:00080000:3.0:1775766495.413243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766495.413251:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066419456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766495.413253:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766495.413263:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775766495.413269:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766495.413276:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766495.413280:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066419840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766495.413281:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066419456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766495.413297:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766495.413300:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766495.413302:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766495.413307:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419584:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775766495.413308:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3326 00000100:00080000:3.0:1775766495.413316:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766495.413321:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766495.413327:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766495.413333:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766495.413341:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066419968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766495.413346:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766495/real 1775766495] req@ffff9d8188741f80 x1862023066419584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766495.413369:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066419584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766495.413376:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766495.413378:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766495.413381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766495.413386:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766495.413399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766495.413405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766495/real 1775766495] req@ffff9d8188742300 x1862023066419712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766495.413413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066419712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766495.413419:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766495.413422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766495.413425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766495.413431:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766495.413467:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766495.413479:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766495/real 1775766495] req@ffff9d8188740700 x1862023066419840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766495.413493:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066419840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766495.413503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766495.413506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766495.413509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766495.413543:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766495.413566:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766495.413593:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766495/real 1775766495] req@ffff9d8188742a00 x1862023066419968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766495.413614:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066419968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766495.413620:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766495.413622:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766495.413624:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066419968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766496.053047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766496.436962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766497.077122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766497.461027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766498.101083:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766498.485021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766499.125038:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766499.508951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766500.149252:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766500.532977:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766500.532986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766500.532991:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766500.532999:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766500.533013:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3332 00000100:00080000:3.0:1775766500.533027:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766500.533036:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766500.533074:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066420096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766500.533120:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766500.533124:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766500.533126:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766500.533130:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3332 00000100:00080000:3.0:1775766500.533133:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766500.533137:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766500.533168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766500.533182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066420224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766500.533193:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766500.533199:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066420096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766500.533199:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766500.533201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766500.533210:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3332 00000100:00080000:3.0:1775766500.533214:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766500.533217:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766500.533221:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766500.533233:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066420352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766500.533242:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766500.533246:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766500.533247:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766500.533250:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3332 00000100:00080000:3.0:1775766500.533266:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766500.533272:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766500.533277:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766500.533282:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766500.533295:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066420480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766500.533303:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766500.533306:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766500.533308:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766500.533312:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3332 00000100:00080000:3.0:1775766500.533317:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766500.533320:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766500.533326:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766500/real 1775766500] req@ffff9d8188741c00 x1862023066420096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775766500.533327:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766500.533342:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066420608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766500.533352:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066420096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766500.533362:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766500.533368:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766500.533373:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066420096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766500.533394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766500.533414:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766500.533424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766500/real 1775766500] req@ffff9d8188740380 x1862023066420224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766500.533435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066420224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766500.533442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766500.533445:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766500.533448:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766500.533452:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766500.533465:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766500.533472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766500/real 1775766500] req@ffff9d8188740000 x1862023066420352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766500.533481:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066420352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766500.533488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766500.533490:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766500.533492:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766500.533497:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766500.533510:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766500.533517:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766500/real 1775766500] req@ffff9d8188742a00 x1862023066420480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766500.533526:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066420480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766500.533559:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766500.533562:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766500.533564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766500.533569:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766500.533585:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766500.533592:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766500/real 1775766500] req@ffff9d8188740700 x1862023066420608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766500.533602:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066420608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766500.533610:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766500.533613:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766500.533615:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766501.173057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766501.557048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766502.197040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766502.581006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766503.221152:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766503.605012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766504.245060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766504.629028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766505.269057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766505.652966:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766505.652968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766505.652983:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766505.652988:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766505.652997:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3337 00000100:00080000:3.0:1775766505.653008:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766505.653016:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766505.653050:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066420736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766505.653075:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766505.653077:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766505.653085:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766505.653102:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3337 00000100:00080000:3.0:1775766505.653105:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766505.653108:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766505.653116:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766505.653125:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066420864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766505.653133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766505.653135:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766505.653137:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766505.653140:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3337 00000100:00080000:3.0:1775766505.653142:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766505.653144:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766505.653147:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766505.653150:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066420736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766505.653156:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066420992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766505.653163:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766505.653165:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766505.653166:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766505.653168:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3337 00000100:00080000:3.0:1775766505.653171:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766505.653173:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766505.653176:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766505.653183:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066421120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766505.653189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766505.653191:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766505.653192:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766505.653194:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3337 00000100:00080000:3.0:1775766505.653203:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766505.653206:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766505.653209:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766505.653215:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766505.653218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066421248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766505.653275:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766505/real 1775766505] req@ffff9d8188742300 x1862023066420736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766505.653292:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066420736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766505.653300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766505.653303:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766505.653306:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066420736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766505.653312:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766505.653329:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766505.653337:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766505/real 1775766505] req@ffff9d8188741f80 x1862023066420864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766505.653345:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066420864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766505.653350:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766505.653352:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766505.653354:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766505.653370:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766505.653380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766505.653386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766505/real 1775766505] req@ffff9d8188743800 x1862023066420992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766505.653393:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066420992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766505.653398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766505.653400:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766505.653402:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066420992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766505.653406:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766505.653416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766505.653421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766505/real 1775766505] req@ffff9d8188742680 x1862023066421120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766505.653428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066421120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766505.653434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766505.653435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766505.653437:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766505.653440:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766505.653449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766505.653454:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766505/real 1775766505] req@ffff9d8188741880 x1862023066421248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766505.653461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066421248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766505.653466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766505.653468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766505.653470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766506.293046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766506.676997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766507.317137:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766507.700963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766508.340969:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766508.724915:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766509.364989:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766509.748978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766510.389113:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766510.772968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766510.772997:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766510.773057:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766510.773061:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766510.773071:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3342 00000100:00080000:3.0:1775766510.773082:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766510.773103:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766510.773137:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066421376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766510.773165:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766510.773170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766510.773171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766510.773175:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3342 00000100:00080000:3.0:1775766510.773178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766510.773181:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766510.773188:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766510.773198:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066421504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766510.773208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766510.773211:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066421376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766510.773212:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766510.773215:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766510.773224:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3342 00000100:00080000:3.0:1775766510.773227:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766510.773230:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766510.773234:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766510.773243:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066421632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766510.773250:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766510.773253:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766510.773254:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766510.773258:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3342 00000100:00080000:3.0:1775766510.773271:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766510.773274:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766510.773279:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766510.773285:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766510.773291:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066421760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766510.773299:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766510.773301:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766510.773303:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766510.773307:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3342 00000100:00080000:3.0:1775766510.773310:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766510.773312:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766510.773316:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766510.773325:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066421888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766510.773343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766510/real 1775766510] req@ffff9d8188741180 x1862023066421376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766510.773365:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066421376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766510.773376:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766510.773381:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766510.773385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066421376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766510.773429:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766510.773456:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766510.773477:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766510/real 1775766510] req@ffff9d8188743100 x1862023066421504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766510.773491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066421504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766510.773500:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766510.773504:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766510.773507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766510.773514:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766510.773536:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766510.773557:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766510/real 1775766510] req@ffff9d8188740e00 x1862023066421632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766510.773570:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066421632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766510.773580:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766510.773583:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766510.773586:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766510.773592:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766510.773610:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766510.773620:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766510/real 1775766510] req@ffff9d8188740a80 x1862023066421760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766510.773642:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066421760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766510.773652:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766510.773656:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766510.773659:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766510.773665:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766510.773681:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766510.773691:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766510/real 1775766510] req@ffff9d8188741500 x1862023066421888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766510.773710:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066421888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766510.773719:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766510.773722:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766510.773725:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066421888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766511.412990:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766511.796985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766512.436975:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766512.820997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766513.461134:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766513.844996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766514.485144:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766514.868971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766515.509084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766515.893020:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766515.893033:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766515.893039:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766515.893049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766515.893052:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3347 00000100:00080000:3.0:1775766515.893071:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766515.893081:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766515.893131:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066422016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766515.893164:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766515.893168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766515.893170:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766515.893174:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3347 00000100:00080000:3.0:1775766515.893178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766515.893182:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766515.893194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766515.893210:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066422016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766515.893210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066422144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766515.893227:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766515.893231:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766515.893233:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766515.893238:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3347 00000100:00080000:3.0:1775766515.893241:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766515.893244:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766515.893249:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766515.893263:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066422272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766515.893272:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766515.893276:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766515.893278:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766515.893282:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3347 00000100:00080000:3.0:1775766515.893285:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766515.893289:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766515.893294:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766515.893305:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066422400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766515.893321:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766515.893325:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766515.893326:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766515.893330:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3347 00000100:00080000:3.0:1775766515.893334:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766515.893337:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766515.893342:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766515.893352:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066422528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766515.894411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766515.894473:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766515/real 1775766515] req@ffff9d8188743480 x1862023066422016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766515.894498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066422016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766515.894510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766515.894515:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766515.894519:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066422016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766515.894557:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766515.894581:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766515.894590:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766515/real 1775766515] req@ffff9d8188742d80 x1862023066422144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766515.894599:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066422144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766515.894605:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766515.894608:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766515.894610:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766515.894614:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766515.894636:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766515.894643:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766515/real 1775766515] req@ffff9d8188743b80 x1862023066422272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766515.894651:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066422272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766515.894656:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766515.894658:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766515.894660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766515.894665:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766515.894677:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766515.894682:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766515/real 1775766515] req@ffff9d8188741500 x1862023066422400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766515.894689:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066422400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766515.894701:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766515.894703:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766515.894705:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766515.894708:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766515.894718:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766515.894724:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766515/real 1775766515] req@ffff9d8188740a80 x1862023066422528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766515.894731:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066422528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766515.894736:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766515.894738:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766515.894740:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766516.533118:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766516.916967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766517.557085:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766517.940969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766518.581077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766518.964959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766519.605057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766519.988994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766520.629059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766521.012981:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766521.012992:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775766521.012996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766521.012996:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766521.013009:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3352 00000100:00080000:3.0:1775766521.013021:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766521.013028:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766521.013065:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066422656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766521.013094:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766521.013098:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766521.013100:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766521.013104:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3352 00000100:00080000:3.0:1775766521.013107:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766521.013111:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766521.013122:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766521.013137:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066422656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766521.013139:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066422784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766521.013163:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766521.013166:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766521.013168:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766521.013173:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3352 00000100:00080000:3.0:1775766521.013186:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766521.013190:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766521.013193:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766521.013201:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766521.013216:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066422912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766521.013226:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766521.013229:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766521.013231:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766521.013235:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3352 00000100:00080000:3.0:1775766521.013238:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766521.013242:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766521.013247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766521.013257:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066423040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766521.013268:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766521.013271:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766521.013274:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766521.013280:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3352 00000100:00080000:3.0:1775766521.013284:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766521.013291:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766521.013293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766521/real 1775766521] req@ffff9d8188740e00 x1862023066422656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775766521.013297:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766521.013314:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066423168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766521.013323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066422656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766521.013337:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766521.013343:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766521.013348:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066422656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766521.013373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766521.013399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766521.013414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766521/real 1775766521] req@ffff9d8188743100 x1862023066422784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766521.013427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066422784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766521.013435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766521.013439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766521.013442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766521.013449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766521.013473:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766521.013485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766521/real 1775766521] req@ffff9d8188741180 x1862023066422912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766521.013498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066422912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766521.013507:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766521.013511:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766521.013515:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066422912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766521.013522:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766521.013545:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766521.013557:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766521/real 1775766521] req@ffff9d8188741880 x1862023066423040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766521.013569:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066423040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766521.013579:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766521.013582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766521.013586:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766521.013592:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766521.013614:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766521.013625:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766521/real 1775766521] req@ffff9d8188742680 x1862023066423168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766521.013638:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066423168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766521.013647:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766521.013651:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766521.013654:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766521.653046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766522.037012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766522.677066:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766523.060986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766523.701035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766524.085025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766524.725173:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766525.108946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766525.749096:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766526.132969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766526.133074:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766526.133079:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766526.133082:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766526.133090:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3357 00000100:00080000:1.0:1775766526.133098:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766526.133104:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766526.133137:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066423296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766526.133165:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766526.133168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766526.133172:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766526.133174:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3357 00000100:00080000:1.0:1775766526.133177:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766526.133179:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766526.133187:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766526.133200:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066423424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766526.133211:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766526.133215:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766526.133217:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766526.133223:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066423296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766526.133224:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3357 00000100:00080000:1.0:1775766526.133232:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766526.133238:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766526.133244:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766526.133258:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066423552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766526.133270:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766526.133274:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766526.133276:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766526.133280:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3357 00000100:00080000:1.0:1775766526.133294:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766526.133298:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766526.133304:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766526.133316:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066423680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766526.133325:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766526.133329:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766526.133331:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766526.133336:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3357 00000100:00080000:1.0:1775766526.133340:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766526.133345:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766526.133351:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766526.133364:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066423808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766526.133365:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766526.133408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766526.133422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766526.133436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766526.773111:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775766527.156935:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766527.797081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766528.181034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766528.821084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766529.204945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775766529.205096:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775766529.205129:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775766529.205142:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766529.205181:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766529.205197:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766529.205212:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766529.205222:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766529.205258:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766526/real 1775766529] req@ffff9d8190553100 x1862023066423552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766529.205294:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066423552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766529.205308:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766529.205314:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766529.205319:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766529.205330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766526/real 1775766529] req@ffff9d8190553480 x1862023066423680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766529.205345:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066423680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766529.205362:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766529.205366:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766529.205370:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766529.205378:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766526/real 1775766529] req@ffff9d8190551880 x1862023066423808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766529.205393:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066423808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766529.205401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766529.205405:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766529.205408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423808:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766529.205417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766526/real 1775766529] req@ffff9d8190551500 x1862023066423296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766529.205430:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066423296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766529.205438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766529.205442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766529.205445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066423296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766529.205453:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766526/real 1775766529] req@ffff9d8190550e00 x1862023066423424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766529.205465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066423424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766529.205474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766529.205478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766529.205481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066423424:192.168.202.105@tcp:38:kworker.0 00010000:00010000:0.0:1775766529.845072:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766530.228979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766530.869021:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766531.252960:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766531.252966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766531.252975:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766531.252981:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766531.252995:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3362 00000100:00080000:1.0:1775766531.253006:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766531.253014:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766531.253050:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066423936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766531.253072:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766531.253075:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766531.253077:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766531.253080:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3362 00000100:00080000:1.0:1775766531.253083:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766531.253085:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766531.253093:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766531.253103:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066424064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766531.253110:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066423936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766531.253114:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766531.253118:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766531.253119:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766531.253123:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3362 00000100:00080000:1.0:1775766531.253125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766531.253127:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766531.253130:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766531.253144:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066424192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766531.253151:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766531.253153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766531.253154:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766531.253157:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3362 00000100:00080000:1.0:1775766531.253165:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766531.253168:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766531.253170:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766531.253177:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766531.253180:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066424320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766531.253187:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766531.253190:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766531.253191:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766531.253193:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3362 00000100:00080000:1.0:1775766531.253196:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766531.253199:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766531.253205:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766531.253212:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066424448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766531.253244:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766531/real 1775766531] req@ffff9d8190550e00 x1862023066423936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766531.253265:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066423936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766531.253277:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766531.253281:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766531.253285:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066423936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766531.253309:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766531.253335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766531.253348:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766531/real 1775766531] req@ffff9d8190551500 x1862023066424064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766531.253360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066424064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766531.253368:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766531.253372:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766531.253374:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766531.253381:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766531.253400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766531.253411:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766531/real 1775766531] req@ffff9d8190551880 x1862023066424192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766531.253424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066424192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766531.253433:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766531.253436:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766531.253439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766531.253446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766531.253468:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766531.253479:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766531/real 1775766531] req@ffff9d8190553480 x1862023066424320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766531.253490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066424320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766531.253499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766531.253502:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766531.253504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766531.253510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766531.253530:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766531.253539:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766531/real 1775766531] req@ffff9d8190553100 x1862023066424448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766531.253549:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066424448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766531.253557:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766531.253560:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766531.253562:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766531.893007:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766532.276987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766532.917043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766533.300949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766533.941007:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766534.324978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766534.965071:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766535.348963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766535.989059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766536.372959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766536.372989:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766536.372995:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766536.372999:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766536.373006:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3367 00000100:00080000:0.0:1775766536.373015:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766536.373020:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766536.373053:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066424576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766536.373081:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766536.373085:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766536.373087:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766536.373091:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3367 00000100:00080000:0.0:1775766536.373095:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766536.373100:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766536.373111:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766536.373127:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066424576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766536.373128:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066424704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766536.373142:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766536.373144:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766536.373146:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766536.373149:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3367 00000100:00080000:0.0:1775766536.373152:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766536.373156:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766536.373161:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766536.373170:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066424832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766536.373176:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766536.373178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766536.373180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766536.373182:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3367 00000100:00080000:0.0:1775766536.373185:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766536.373188:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766536.373192:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766536.373196:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766536.373205:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066424960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766536.373212:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766536.373214:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766536.373216:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766536.373219:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3367 00000100:00080000:0.0:1775766536.373221:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766536.373224:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766536.373228:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766536.373233:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766536/real 1775766536] req@ffff9d81901af100 x1862023066424576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766536.373237:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066425088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766536.373273:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066424576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766536.373282:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766536.373286:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766536.373289:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066424576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766536.373306:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766536.373322:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766536.373330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766536/real 1775766536] req@ffff9d81901afb80 x1862023066424704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766536.373338:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066424704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766536.373343:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766536.373345:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766536.373347:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766536.373351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766536.373360:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766536.373365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766536/real 1775766536] req@ffff9d81901aed80 x1862023066424832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766536.373373:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066424832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766536.373378:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766536.373379:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766536.373381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766536.373384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766536.373394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766536.373399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766536/real 1775766536] req@ffff9d81901ae680 x1862023066424960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766536.373411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066424960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766536.373417:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766536.373419:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766536.373421:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066424960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766536.373424:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766536.373434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766536.373439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766536/real 1775766536] req@ffff9d81901ad180 x1862023066425088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766536.373446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066425088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766536.373451:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766536.373453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766536.373455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766537.013036:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766537.397007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766538.037060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766538.420999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766539.061126:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766539.444949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766540.085088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766540.468935:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766541.109111:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766541.492953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766541.493220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766541.493263:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766541.493270:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766541.493332:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3373 00000100:00080000:3.0:1775766541.493379:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766541.493406:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766541.493501:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066425216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766541.493569:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766541.493576:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766541.493580:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766541.493609:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066425216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766541.493657:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3373 00000100:00080000:3.0:1775766541.493675:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766541.493681:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766541.493683:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766541.493694:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766541.493733:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066425344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766541.493742:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766541/real 1775766541] req@ffff9d8188743800 x1862023066425216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766541.493748:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766541.493755:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766541.493757:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766541.493762:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3373 00000100:00100000:0.0:1775766541.493766:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066425216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766541.493766:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766541.493773:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766541.493779:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1775766541.493783:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766541.493788:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766541.493791:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066425472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766541.493795:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066425216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766541.493808:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766541.493811:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766541.493812:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766541.493815:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3373 00000100:00080000:3.0:1775766541.493818:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766541.493820:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766541.493823:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766541.493831:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066425600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766541.493833:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425344:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775766541.493847:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766541.493850:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766541.493851:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766541.493854:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3373 00000100:00080000:3.0:1775766541.493859:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766541.493863:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766541.493863:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766541.493869:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766541.493896:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766541/real 1775766541] req@ffff9d8188741f80 x1862023066425344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766541.493906:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066425344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766541.493910:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066425728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766541.493927:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766541.493932:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766541.493936:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766541.493942:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766541.493956:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766541.493963:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766541/real 1775766541] req@ffff9d8188742300 x1862023066425472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766541.493971:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066425472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766541.493976:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766541.493978:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766541.493980:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766541.493989:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766541.494000:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766541.494006:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766541/real 1775766541] req@ffff9d8188740700 x1862023066425600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766541.494014:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066425600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766541.494020:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766541.494022:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766541.494024:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766541.494027:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766541.494036:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766541.494041:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766541/real 1775766541] req@ffff9d8188742a00 x1862023066425728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766541.494048:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066425728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766541.494053:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766541.494055:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766541.494057:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766542.133256:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766542.517003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766543.157077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766543.541000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766544.181160:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766544.565000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766545.205095:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766545.588996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766546.229098:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766546.612956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766546.612972:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766546.612978:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766546.612981:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766546.612990:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3378 00000100:00080000:1.0:1775766546.612999:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766546.613005:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766546.613037:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066425856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766546.613059:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766546.613062:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766546.613063:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766546.613065:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3378 00000100:00080000:1.0:1775766546.613068:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766546.613076:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766546.613086:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766546.613094:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066425984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766546.613101:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766546.613103:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766546.613104:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766546.613106:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3378 00000100:00080000:1.0:1775766546.613109:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766546.613111:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766546.613114:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766546.613120:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066426112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766546.613127:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766546.613131:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775766546.613132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066425856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766546.613133:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766546.613140:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3378 00000100:00080000:1.0:1775766546.613143:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766546.613145:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766546.613149:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766546.613156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066426240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766546.613161:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766546.613163:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766546.613164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766546.613166:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3378 00000100:00080000:1.0:1775766546.613168:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766546.613170:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766546.613172:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766546.613177:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066426368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766546.613250:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766546.613288:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766546/real 1775766546] req@ffff9d8190553100 x1862023066425856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766546.613301:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066425856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766546.613309:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766546.613312:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766546.613315:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066425856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766546.613320:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766546.613333:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766546.613340:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766546/real 1775766546] req@ffff9d8190553480 x1862023066425984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766546.613348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066425984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766546.613353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766546.613354:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766546.613356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066425984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766546.613373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766546.613383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766546.613388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766546/real 1775766546] req@ffff9d8190551880 x1862023066426112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766546.613396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066426112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766546.613405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766546.613407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766546.613409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766546.613412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766546.613422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766546.613427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766546/real 1775766546] req@ffff9d8190551500 x1862023066426240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766546.613434:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066426240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766546.613440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766546.613442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766546.613444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766546.613447:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766546.613456:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766546.613461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766546/real 1775766546] req@ffff9d8190550e00 x1862023066426368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766546.613467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066426368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766546.613472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766546.613474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766546.613476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766547.253051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766547.636985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766548.277170:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766548.661105:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766549.301131:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766549.685018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766550.325154:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766550.708970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766551.349094:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766551.732983:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766551.732990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766551.732993:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766551.733000:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766551.733011:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3383 00000100:00080000:1.0:1775766551.733024:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766551.733034:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766551.733096:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066426496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766551.733135:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766551.733144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766551.733146:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766551.733151:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3383 00000100:00080000:1.0:1775766551.733157:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766551.733162:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766551.733173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766551.733193:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066426624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766551.733205:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066426496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766551.733214:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766551.733221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766551.733223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766551.733230:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3383 00000100:00080000:1.0:1775766551.733236:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766551.733241:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766551.733250:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766551.733269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066426752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766551.733282:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766551.733286:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766551.733290:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766551.733293:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766551.733298:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3383 00000100:00080000:1.0:1775766551.733305:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766551.733308:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766551.733315:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766551.733325:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066426880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766551.733333:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766551.733336:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766551.733338:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766551.733340:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3383 00000100:00080000:1.0:1775766551.733343:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766551.733345:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766551.733349:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766551.733356:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066427008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766551.733439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766551/real 1775766551] req@ffff9d8190550e00 x1862023066426496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766551.733475:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066426496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766551.733492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766551.733496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766551.733501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066426496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766551.733525:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766551.733552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766551.733586:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766551/real 1775766551] req@ffff9d8190551500 x1862023066426624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766551.733603:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066426624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766551.733644:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766551.733649:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766551.733654:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766551.733680:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766551.733702:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766551.733712:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766551/real 1775766551] req@ffff9d8190551880 x1862023066426752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766551.733724:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066426752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766551.733732:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766551.733735:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766551.733748:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766551.733753:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766551.733767:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766551.733774:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766551/real 1775766551] req@ffff9d8190553480 x1862023066426880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766551.733785:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066426880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766551.733793:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766551.733796:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766551.733799:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066426880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766551.733803:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766551.733826:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766551.733834:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766551/real 1775766551] req@ffff9d8190553100 x1862023066427008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766551.733844:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066427008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766551.733852:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766551.733854:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766551.733857:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766552.373192:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766552.756954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766553.397166:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766553.780962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766554.421021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766554.804958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766555.445198:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766555.829004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766556.469012:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766556.852943:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766556.852951:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766556.852955:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766556.852957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766556.852963:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3388 00000100:00080000:1.0:1775766556.852973:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766556.852979:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766556.853019:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066427136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766556.853045:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766556.853048:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766556.853050:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766556.853055:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3388 00000100:00080000:1.0:1775766556.853061:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766556.853066:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766556.853077:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066427136:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775766556.853077:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766556.853098:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066427264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766556.853112:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766556.853115:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766556.853117:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766556.853122:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3388 00000100:00080000:1.0:1775766556.853126:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766556.853132:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766556.853133:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766556.853141:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766556.853156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066427392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766556.853171:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766556/real 1775766556] req@ffff9d8190553100 x1862023066427136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766556.853174:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766556.853184:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766556.853186:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766556.853193:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066427136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766556.853194:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3388 00000100:00080000:1.0:1775766556.853203:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766556.853205:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766556.853210:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775766556.853211:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766556.853216:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066427136:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775766556.853220:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766556.853236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066427520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766556.853237:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427264:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775766556.853256:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766556.853259:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766556.853260:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766556.853265:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3388 00000800:00000100:0.0:1775766556.853269:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766556.853269:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766556.853274:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766556.853280:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766556.853283:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766556/real 1775766556] req@ffff9d8190553480 x1862023066427264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766556.853297:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066427648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766556.853299:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066427264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766556.853327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766556.853331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766556.853336:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766556.853340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766556.853353:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766556.853359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766556/real 1775766556] req@ffff9d8190551880 x1862023066427392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766556.853367:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066427392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766556.853372:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766556.853373:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766556.853375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766556.853383:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766556.853394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766556.853399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766556/real 1775766556] req@ffff9d8190551500 x1862023066427520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766556.853406:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066427520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766556.853412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766556.853414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766556.853416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766556.853420:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766556.853428:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766556.853433:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766556/real 1775766556] req@ffff9d8190550e00 x1862023066427648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766556.853440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066427648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766556.853445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766556.853447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766556.853448:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766557.493067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766557.876939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766558.517020:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766558.901013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766559.541081:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766559.924945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766560.565062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766560.948962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766561.589083:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766561.972972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766561.973010:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766561.973019:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766561.973024:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766561.973036:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3393 00000100:00080000:0.0:1775766561.973047:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766561.973054:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766561.973097:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066427776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766561.973130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766561.973134:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766561.973136:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766561.973140:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3393 00000100:00080000:0.0:1775766561.973143:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766561.973146:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766561.973156:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766561.973167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066427904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766561.973177:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766561.973180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766561.973181:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766561.973185:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3393 00000100:00080000:0.0:1775766561.973187:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766561.973191:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766561.973195:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766561.973204:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066428032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766561.973214:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766561.973217:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766561.973219:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766561.973224:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3393 00000100:00080000:0.0:1775766561.973231:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766561.973231:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066427776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766561.973238:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766561.973245:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766561.973256:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066428160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766561.973267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766561.973271:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766561.973273:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766561.973276:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3393 00000100:00080000:0.0:1775766561.973312:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766561.973318:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775766561.973324:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775766561.973331:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766561.973355:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066428288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766561.973374:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766561/real 1775766561] req@ffff9d81901ad180 x1862023066427776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766561.973396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066427776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766561.973407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766561.973411:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766561.973416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066427776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766561.973423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766561.973442:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766561.973453:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766561/real 1775766561] req@ffff9d81901ae680 x1862023066427904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766561.973465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066427904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766561.973472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766561.973476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766561.973479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066427904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766561.973484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766561.973548:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766561.973566:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766561/real 1775766561] req@ffff9d81901aed80 x1862023066428032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766561.973586:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066428032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766561.973599:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766561.973604:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766561.973609:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766561.973638:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766561.973664:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766561.973680:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766561/real 1775766561] req@ffff9d81901afb80 x1862023066428160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766561.973699:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066428160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766561.973713:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766561.973718:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766561.973723:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766561.973732:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766561.973759:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766561.973772:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766561/real 1775766561] req@ffff9d81901af100 x1862023066428288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766561.973784:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066428288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766561.973792:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766561.973795:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766561.973798:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766562.613143:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766562.997004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766563.637071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766564.020942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766564.661088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766565.045053:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766565.685118:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766566.068978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766566.709053:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766567.093018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766567.093023:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766567.093029:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766567.093033:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766567.093041:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3398 00000100:00080000:3.0:1775766567.093050:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766567.093055:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766567.093087:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066428416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766567.093111:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766567.093114:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766567.093115:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766567.093118:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3398 00000100:00080000:3.0:1775766567.093120:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766567.093122:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766567.093130:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766567.093138:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066428544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766567.093146:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766567.093149:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766567.093151:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766567.093155:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066428416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766567.093156:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3398 00000100:00080000:3.0:1775766567.093159:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766567.093165:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766567.093169:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766567.093176:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066428672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766567.093181:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766567.093184:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766567.093185:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766567.093188:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3398 00000100:00080000:3.0:1775766567.093190:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766567.093192:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766567.093195:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766567.093204:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066428800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766567.093209:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766567.093211:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766567.093213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766567.093215:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3398 00000100:00080000:3.0:1775766567.093217:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766567.093219:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766567.093222:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766567.093227:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766567.093233:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066428928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766567.093277:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766567/real 1775766567] req@ffff9d8188740000 x1862023066428416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766567.093306:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066428416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766567.093316:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766567.093321:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766567.093340:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066428416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766567.093363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766567.093412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766567.093427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766567/real 1775766567] req@ffff9d8188740380 x1862023066428544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766567.093441:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066428544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766567.093484:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766567.093488:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766567.093492:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766567.093498:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766567.093521:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766567.093534:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766567/real 1775766567] req@ffff9d8188741c00 x1862023066428672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766567.093547:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066428672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766567.093563:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766567.093577:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766567.093580:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766567.093587:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766567.093610:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766567.093621:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766567/real 1775766567] req@ffff9d8188742a00 x1862023066428800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766567.093635:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066428800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766567.093644:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766567.093648:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766567.093652:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766567.093657:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766567.093676:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766567.093702:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766567/real 1775766567] req@ffff9d8188740700 x1862023066428928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766567.093716:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066428928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766567.093725:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766567.093729:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766567.093733:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066428928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766567.733023:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766568.116958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766568.757126:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766569.140993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766569.781108:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766570.164964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766570.805055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766571.188998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766571.829099:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766572.213013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766572.213062:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766572.213079:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766572.213086:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766572.213100:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3403 00000100:00080000:3.0:1775766572.213123:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766572.213133:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766572.213198:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066429056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766572.213246:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766572.213254:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766572.213257:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766572.213264:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3403 00000100:00080000:3.0:1775766572.213270:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766572.213276:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766572.213290:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766572.213324:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066429184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766572.213340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066429056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766572.213351:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766572.213359:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766572.213362:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766572.213372:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3403 00000100:00080000:3.0:1775766572.213380:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766572.213385:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766572.213409:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766572.213452:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066429312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766572.213465:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766572.213469:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766572.213472:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766572.213477:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3403 00000100:00080000:3.0:1775766572.213484:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766572.213495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766572.213500:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766572.213510:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766572.213547:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066429440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766572.213558:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766572.213562:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766572.213565:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766572.213571:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3403 00000100:00080000:3.0:1775766572.213578:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766572.213585:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766572.213594:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766572/real 1775766572] req@ffff9d8188742300 x1862023066429056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775766572.213600:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766572.213655:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066429568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766572.213656:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066429056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766572.213753:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766572.213764:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766572.213781:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066429056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766572.213815:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766572.213860:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766572.213917:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766572/real 1775766572] req@ffff9d8188741f80 x1862023066429184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766572.213936:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066429184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766572.213946:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766572.213951:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766572.213956:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766572.213969:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766572.214014:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766572.214034:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766572/real 1775766572] req@ffff9d8188743800 x1862023066429312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766572.214058:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066429312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766572.214074:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766572.214079:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766572.214083:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766572.214092:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766572.214120:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766572.214133:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766572/real 1775766572] req@ffff9d8188742680 x1862023066429440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766572.214147:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066429440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766572.214157:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766572.214161:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766572.214164:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766572.214171:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766572.214188:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766572.214198:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766572/real 1775766572] req@ffff9d8188741880 x1862023066429568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766572.214211:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066429568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766572.214221:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766572.214224:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766572.214228:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766572.853095:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766573.236999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766573.877079:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766574.260948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766574.901131:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766575.284959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766575.925020:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766576.309024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766576.949107:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766577.333016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766577.333083:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766577.333098:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766577.333104:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766577.333120:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3408 00000100:00080000:0.0:1775766577.333136:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766577.333145:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766577.333209:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066429696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766577.333252:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766577.333256:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766577.333258:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766577.333261:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3408 00000100:00080000:0.0:1775766577.333264:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766577.333269:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766577.333282:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766577.333299:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066429824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766577.333314:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766577.333318:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066429696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766577.333321:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766577.333326:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766577.333334:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3408 00000100:00080000:0.0:1775766577.333361:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766577.333367:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766577.333375:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766577.333389:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066429952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766577.333398:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766577.333402:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766577.333404:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766577.333407:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3408 00000100:00080000:0.0:1775766577.333434:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766577.333438:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766577.333442:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766577.333448:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766577.333454:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066430080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766577.333462:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766577.333466:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766577.333468:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766577.333471:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3408 00000100:00080000:0.0:1775766577.333476:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766577.333479:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766577.333482:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766577.333491:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066430208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766577.333525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766577/real 1775766577] req@ffff9d81901af100 x1862023066429696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766577.333547:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066429696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766577.333556:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766577.333560:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766577.333564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066429696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766577.333591:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766577.333620:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766577.333636:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766577/real 1775766577] req@ffff9d81901afb80 x1862023066429824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766577.333653:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066429824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766577.333677:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766577.333681:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766577.333683:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766577.333689:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766577.333708:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766577.333722:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766577/real 1775766577] req@ffff9d81901aed80 x1862023066429952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766577.333737:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066429952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766577.333748:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766577.333752:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766577.333757:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066429952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766577.333780:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766577.333803:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766577.333812:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766577/real 1775766577] req@ffff9d81901ae680 x1862023066430080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766577.333828:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066430080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766577.333836:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766577.333839:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766577.333842:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766577.333846:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766577.333859:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766577.333865:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766577/real 1775766577] req@ffff9d81901ad180 x1862023066430208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766577.333873:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066430208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766577.333880:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766577.333882:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766577.333884:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766577.973108:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766578.357002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766578.997085:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766579.380968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766580.021102:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766580.405023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766581.045112:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766581.428961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766582.069076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766582.453052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766582.453065:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766582.453081:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766582.453090:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766582.453109:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3413 00000100:00080000:0.0:1775766582.453130:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766582.453142:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766582.453238:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066430336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766582.453305:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766582.453312:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766582.453332:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766582.453338:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3413 00000100:00080000:0.0:1775766582.453344:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766582.453352:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766582.453369:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766582.453390:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066430464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766582.453404:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766582.453412:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766582.453412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066430336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766582.453418:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766582.453430:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3413 00000100:00080000:0.0:1775766582.453436:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766582.453442:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766582.453451:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766582.453473:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066430592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766582.453497:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766582.453501:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766582.453503:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766582.453508:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3413 00000800:00000100:3.0:1775766582.453514:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766582.453517:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766582.453524:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766582.453531:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766582.453546:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066430720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766582.453562:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766582.453565:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766582.453566:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766582.453570:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3413 00000100:00080000:0.0:1775766582.453574:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766582.453577:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766582.453582:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766582.453586:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766582/real 1775766582] req@ffff9d81901ad180 x1862023066430336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766582.453597:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066430848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766582.453615:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066430336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766582.453636:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766582.453641:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766582.453645:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066430336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766582.453672:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766582.453694:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766582.453705:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766582/real 1775766582] req@ffff9d81901ae680 x1862023066430464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766582.453717:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066430464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766582.453725:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766582.453728:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766582.453731:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766582.453736:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766582.453750:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766582.453757:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766582/real 1775766582] req@ffff9d81901aed80 x1862023066430592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766582.453767:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066430592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766582.453775:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766582.453777:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766582.453780:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766582.453786:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766582.453800:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766582.453807:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766582/real 1775766582] req@ffff9d81901afb80 x1862023066430720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766582.453817:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066430720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766582.453826:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766582.453829:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766582.453831:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766582.453835:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766582.453849:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766582.453856:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766582/real 1775766582] req@ffff9d81901af100 x1862023066430848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766582.453866:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066430848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766582.453928:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766582.453933:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766582.453936:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066430848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766583.093203:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766583.477027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766584.117046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766584.501050:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766585.141028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766585.524967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766586.165010:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766586.548947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766587.189061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766587.572966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766587.572997:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766587.573004:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766587.573008:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766587.573020:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3419 00000100:00080000:1.0:1775766587.573032:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766587.573042:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766587.573088:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066430976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766587.573122:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766587.573125:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766587.573127:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766587.573130:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3419 00000100:00080000:1.0:1775766587.573134:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766587.573139:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766587.573148:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766587.573160:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066431104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766587.573170:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766587.573173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766587.573175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766587.573179:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3419 00000100:00080000:1.0:1775766587.573181:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766587.573184:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766587.573188:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766587.573196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066431232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766587.573204:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066430976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766587.573209:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766587.573216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766587.573218:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766587.573223:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3419 00000100:00080000:1.0:1775766587.573226:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766587.573244:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766587.573252:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766587.573270:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066431360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766587.573285:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766587.573290:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766587.573293:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766587.573298:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3419 00000100:00080000:1.0:1775766587.573304:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766587.573307:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766587.573314:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766587.573322:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766587.573338:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066431488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766587.573379:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766587/real 1775766587] req@ffff9d8190550e00 x1862023066430976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766587.573404:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066430976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766587.573416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766587.573421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766587.573426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066430976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766587.573435:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766587.573462:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766587.573476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766587/real 1775766587] req@ffff9d8190551500 x1862023066431104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766587.573492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066431104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766587.573502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766587.573506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766587.573510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766587.573536:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766587.573559:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766587.573571:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766587/real 1775766587] req@ffff9d8190551880 x1862023066431232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766587.573585:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066431232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766587.573603:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766587.573608:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766587.573612:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766587.573619:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766587.573640:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766587.573652:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766587/real 1775766587] req@ffff9d8190553480 x1862023066431360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766587.573668:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066431360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766587.573680:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766587.573684:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766587.573688:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766587.573695:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766587.573717:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766587.573729:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766587/real 1775766587] req@ffff9d8190553100 x1862023066431488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766587.573745:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066431488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766587.573756:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766587.573760:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766587.573765:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766588.213053:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766588.596949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766589.237122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766589.621030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766590.261091:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766590.645036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766591.285032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766591.668962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766592.309253:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766592.692964:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766592.692973:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775766592.692973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766592.692976:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766592.692987:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3424 00000100:00080000:1.0:1775766592.692995:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766592.693001:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766592.693035:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066431616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766592.693061:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766592.693064:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766592.693065:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766592.693068:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3424 00000100:00080000:1.0:1775766592.693071:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766592.693075:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766592.693084:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766592.693098:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066431744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766592.693119:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066431616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766592.693137:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766592.693140:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766592.693141:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766592.693146:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3424 00000100:00080000:1.0:1775766592.693149:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766592.693153:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766592.693158:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766592.693167:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066431872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766592.693174:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766592.693188:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766592.693191:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766592.693195:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3424 00000100:00080000:1.0:1775766592.693199:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766592.693201:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766592.693206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766592.693216:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066432000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766592.693223:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766592.693226:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766592.693227:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766592.693230:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3424 00000100:00080000:1.0:1775766592.693233:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766592.693236:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766592.693240:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766592.693247:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766592.693250:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066432128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766592.693289:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766592.693301:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766592.693316:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766593.333128:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766593.717051:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766594.357095:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766594.740971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766595.381021:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766595.764934:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:0.0:1775766595.765100:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:0.0:1775766595.765129:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775766595.765181:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775766595.765223:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775766595.765232:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775766595.765240:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775766595.765247:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775766595.765294:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766592/real 1775766595] req@ffff9d8190550e00 x1862023066432128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766595.765321:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066432128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766595.765335:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766595.765340:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766595.765344:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432128:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766595.765352:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766592/real 1775766595] req@ffff9d8190553480 x1862023066431744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766595.765364:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066431744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766595.765371:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766595.765375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766595.765380:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766595.765386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766592/real 1775766595] req@ffff9d8190551880 x1862023066431872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766595.765397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066431872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766595.765405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766595.765408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766595.765411:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066431872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766595.765418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766592/real 1775766595] req@ffff9d8190551500 x1862023066432000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766595.765429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066432000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766595.765439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766595.765442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766595.765445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766595.765450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766592/real 1775766595] req@ffff9d8190553100 x1862023066431616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766595.765468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066431616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766595.765475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766595.765477:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766595.765480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066431616:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775766596.405222:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766596.789012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766597.429197:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766597.812981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766597.813142:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766597.813151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766597.813156:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766597.813170:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3429 00000100:00080000:0.0:1775766597.813185:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766597.813197:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766597.813252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066432256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766597.813287:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766597.813294:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766597.813297:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766597.813306:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3429 00000100:00080000:0.0:1775766597.813315:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766597.813322:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766597.813336:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766597.813341:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066432256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766597.813367:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066432384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766597.813390:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766597.813397:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766597.813400:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766597.813408:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3429 00000100:00080000:0.0:1775766597.813440:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766597.813447:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775766597.813457:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775766597.813458:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766597.813487:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066432512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766597.813506:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766597.813512:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766597.813515:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766597.813524:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3429 00000100:00080000:0.0:1775766597.813533:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766597.813538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766597/real 1775766597] req@ffff9d81901af100 x1862023066432256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766597.813550:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766597.813586:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766597.813594:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066432256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766597.813614:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775766597.813619:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066432640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766597.813629:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766597.813649:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066432256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766597.813666:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766597.813674:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766597.813678:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766597.813690:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3429 00000100:00080000:0.0:1775766597.813698:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766597.813707:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432384:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775766597.813708:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766597.813729:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766597.813754:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066432768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766597.813771:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766597.813798:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766597/real 1775766597] req@ffff9d81901afb80 x1862023066432384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766597.813818:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066432384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766597.813827:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766597.813832:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766597.813835:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766597.813843:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766597.813864:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766597.813926:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766597/real 1775766597] req@ffff9d81901aed80 x1862023066432512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766597.813942:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066432512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766597.813952:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766597.813955:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766597.813959:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766597.813966:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766597.813991:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766597.814004:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766597/real 1775766597] req@ffff9d81901ae680 x1862023066432640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766597.814017:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066432640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766597.814034:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766597.814038:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766597.814042:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766597.814063:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766597.814084:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766597.814096:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766597/real 1775766597] req@ffff9d81901ad180 x1862023066432768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766597.814108:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066432768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766597.814140:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766597.814148:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766597.814152:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066432768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766598.453165:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766598.837019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766599.477024:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766599.861069:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766600.501120:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766600.884978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766601.525026:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766601.909013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766602.549189:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766602.933025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766602.933067:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766602.933100:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766602.933107:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766602.933119:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3434 00000100:00080000:1.0:1775766602.933133:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766602.933142:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766602.933200:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066432896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766602.933243:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766602.933247:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766602.933248:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766602.933252:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3434 00000100:00080000:1.0:1775766602.933256:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766602.933260:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766602.933270:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766602.933283:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066433024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766602.933301:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766602.933306:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766602.933308:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766602.933312:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3434 00000100:00080000:1.0:1775766602.933315:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766602.933318:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766602.933321:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766602.933330:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066433152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766602.933340:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766602.933343:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766602.933345:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766602.933349:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3434 00000100:00080000:1.0:1775766602.933352:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766602.933355:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766602.933360:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766602.933363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066432896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766602.933389:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066433280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766602.933399:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766602.933402:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766602.933405:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766602.933408:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3434 00000100:00080000:1.0:1775766602.933412:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766602.933417:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766602.933423:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766602.933435:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066433408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766602.933456:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766602.933504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766602/real 1775766602] req@ffff9d8190553100 x1862023066432896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766602.933523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066432896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766602.933535:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766602.933539:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766602.933543:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066432896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766602.933551:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766602.933570:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766602.933581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766602/real 1775766602] req@ffff9d8190551500 x1862023066433024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766602.933593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066433024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766602.933601:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766602.933604:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766602.933607:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766602.933613:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766602.933628:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766602.933636:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766602/real 1775766602] req@ffff9d8190551880 x1862023066433152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766602.933647:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066433152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766602.933655:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766602.933657:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766602.933660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766602.933693:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766602.933709:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766602.933717:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766602/real 1775766602] req@ffff9d8190553480 x1862023066433280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766602.933727:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066433280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766602.933737:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766602.933740:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766602.933743:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766602.933748:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766602.933762:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766602.933770:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766602/real 1775766602] req@ffff9d8190550e00 x1862023066433408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766602.933781:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066433408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766602.933789:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766602.933792:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766602.933795:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766603.573057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766603.956992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766604.597151:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766604.980994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766605.621146:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766606.005014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766606.645111:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766607.028988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766607.669110:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766608.052994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766608.053068:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766608.053074:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766608.053078:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766608.053086:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3439 00000100:00080000:3.0:1775766608.053096:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766608.053101:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766608.053138:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066433536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766608.053168:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766608.053171:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766608.053173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766608.053176:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3439 00000100:00080000:3.0:1775766608.053178:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766608.053181:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766608.053189:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766608.053203:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066433664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766608.053216:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766608.053219:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766608.053221:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766608.053224:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3439 00000100:00080000:3.0:1775766608.053227:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766608.053230:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766608.053234:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766608.053239:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066433536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766608.053246:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066433792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766608.053254:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766608.053257:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766608.053258:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766608.053261:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3439 00000100:00080000:3.0:1775766608.053264:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766608.053267:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766608.053270:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766608.053280:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066433920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766608.053287:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766608.053290:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766608.053292:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766608.053295:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3439 00000100:00080000:3.0:1775766608.053298:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766608.053302:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766608.053308:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766608.053310:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766608.053322:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066434048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766608.053356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766608/real 1775766608] req@ffff9d8188741180 x1862023066433536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766608.053373:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066433536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766608.053382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766608.053385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766608.053388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066433536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766608.053394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766608.053411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766608.053419:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766608/real 1775766608] req@ffff9d8188743100 x1862023066433664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766608.053429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066433664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766608.053436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766608.053438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766608.053441:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766608.053460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766608.053472:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766608.053478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766608/real 1775766608] req@ffff9d8188740e00 x1862023066433792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766608.053487:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066433792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766608.053493:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766608.053495:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766608.053497:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766608.053501:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766608.053513:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766608.053520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766608/real 1775766608] req@ffff9d8188740a80 x1862023066433920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766608.053528:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066433920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766608.053534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766608.053536:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766608.053539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066433920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766608.053544:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766608.053555:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766608.053560:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766608/real 1775766608] req@ffff9d8188741500 x1862023066434048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766608.053569:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066434048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766608.053575:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766608.053577:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766608.053579:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766608.693033:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766609.077033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766609.716990:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766610.100981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766610.741041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766611.124980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766611.765021:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766612.149114:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766612.789156:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766613.173045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766613.173074:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766613.173088:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766613.173113:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766613.173127:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3444 00000100:00080000:1.0:1775766613.173142:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766613.173152:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766613.173218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066434176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766613.173266:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766613.173271:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766613.173274:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766613.173279:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3444 00000100:00080000:1.0:1775766613.173284:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766613.173288:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766613.173300:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766613.173332:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066434304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766613.173344:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766613.173347:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766613.173349:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766613.173353:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3444 00000100:00080000:1.0:1775766613.173358:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766613.173362:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766613.173367:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766613.173377:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066434432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766613.173388:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766613.173391:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766613.173394:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766613.173398:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3444 00000100:00080000:1.0:1775766613.173402:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766613.173406:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766613.173412:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766613.173418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066434176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766613.173428:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066434560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766613.173440:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766613.173443:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766613.173446:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766613.173450:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3444 00000100:00080000:1.0:1775766613.173454:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766613.173458:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766613.173464:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766613.173477:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066434688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766613.173503:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766613.173558:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766613/real 1775766613] req@ffff9d8190550e00 x1862023066434176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766613.173580:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066434176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766613.173593:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766613.173597:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766613.173603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066434176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766613.173611:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766613.173633:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766613.173646:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766613/real 1775766613] req@ffff9d8190553480 x1862023066434304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766613.173659:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066434304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766613.173668:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766613.173671:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766613.173675:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766613.173681:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766613.173698:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766613.173707:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766613/real 1775766613] req@ffff9d8190551880 x1862023066434432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766613.173720:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066434432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766613.173729:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766613.173732:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766613.173735:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766613.173763:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766613.173780:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766613.173791:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766613/real 1775766613] req@ffff9d8190551500 x1862023066434560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766613.173804:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066434560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766613.173814:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766613.173817:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766613.173820:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766613.173826:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766613.173842:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766613.173850:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766613/real 1775766613] req@ffff9d8190553100 x1862023066434688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766613.173863:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066434688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766613.173937:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766613.173945:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766613.173951:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766613.813111:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766614.197096:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766614.837129:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766615.221008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766615.861063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766616.244985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766616.885116:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766617.268971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766617.909142:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766618.293016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766618.293027:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766618.293036:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766618.293040:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766618.293050:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3449 00000100:00080000:3.0:1775766618.293061:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766618.293068:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766618.293104:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066434816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766618.293129:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766618.293132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766618.293133:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766618.293136:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3449 00000100:00080000:3.0:1775766618.293139:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766618.293141:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766618.293160:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766618.293168:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066434944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766618.293174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766618.293178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766618.293180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766618.293184:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3449 00000100:00080000:3.0:1775766618.293186:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766618.293188:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766618.293191:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766618.293198:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066435072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766618.293203:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766618.293206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766618.293207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766618.293210:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3449 00000100:00080000:3.0:1775766618.293213:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766618.293218:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066434816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766618.293218:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766618.293223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766618.293233:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066435200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766618.293239:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766618.293242:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766618.293244:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766618.293246:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3449 00000100:00080000:3.0:1775766618.293248:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766618.293250:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766618.293254:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766618.293262:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066435328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766618.293287:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766618.293335:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766618/real 1775766618] req@ffff9d8188743b80 x1862023066434816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766618.293355:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066434816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766618.293366:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766618.293371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766618.293376:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066434816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766618.293395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766618.293420:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766618.293445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766618/real 1775766618] req@ffff9d8188742d80 x1862023066434944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766618.293459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066434944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766618.293469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766618.293472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766618.293476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066434944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766618.293483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766618.293504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766618.293515:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766618/real 1775766618] req@ffff9d8188743480 x1862023066435072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766618.293528:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066435072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766618.293543:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766618.293547:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766618.293550:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766618.293577:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766618.293597:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766618.293609:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766618/real 1775766618] req@ffff9d8188741500 x1862023066435200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766618.293622:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066435200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766618.293632:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766618.293636:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766618.293639:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766618.293646:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766618.293665:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766618.293675:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766618/real 1775766618] req@ffff9d8188740a80 x1862023066435328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766618.293688:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066435328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766618.293708:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766618.293711:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766618.293715:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766618.933098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766619.317026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766619.957079:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766620.340974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766620.981039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766621.364967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766622.005062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766622.388973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766623.029017:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766623.412937:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766623.412946:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775766623.412948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766623.412949:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766623.412960:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3454 00000100:00080000:3.0:1775766623.412969:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766623.412976:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766623.413014:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066435456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766623.413047:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766623.413050:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766623.413052:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766623.413055:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3454 00000100:00080000:3.0:1775766623.413058:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766623.413061:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766623.413069:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766623.413080:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066435584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766623.413088:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766623.413090:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766623.413092:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766623.413095:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3454 00000100:00080000:3.0:1775766623.413097:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766623.413102:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766623.413103:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066435456:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775766623.413105:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766623.413116:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066435712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766623.413122:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766623.413125:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766623.413126:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766623.413129:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3454 00000100:00080000:3.0:1775766623.413131:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766623.413133:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766623.413138:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766623.413145:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066435840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766623.413152:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766623.413154:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766623.413155:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766623.413157:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3454 00000100:00080000:3.0:1775766623.413159:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766623.413162:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766623.413167:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766623.413174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766623.413183:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066435968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766623.413221:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766623/real 1775766623] req@ffff9d8188740e00 x1862023066435456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766623.413239:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066435456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766623.413248:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766623.413252:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766623.413256:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066435456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766623.413265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766623.413289:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766623.413302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766623/real 1775766623] req@ffff9d8188743100 x1862023066435584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766623.413312:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066435584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766623.413317:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766623.413319:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766623.413322:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766623.413350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766623.413364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766623.413371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766623/real 1775766623] req@ffff9d8188741180 x1862023066435712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766623.413379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066435712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766623.413384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766623.413386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766623.413388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766623.413392:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766623.413402:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766623.413408:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766623/real 1775766623] req@ffff9d8188741880 x1862023066435840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766623.413415:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066435840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766623.413421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766623.413422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766623.413424:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766623.413427:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766623.413436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766623.413441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766623/real 1775766623] req@ffff9d8188742680 x1862023066435968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766623.413448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066435968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766623.413454:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766623.413455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766623.413457:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066435968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766624.053026:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766624.436940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766625.077035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766625.460953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766626.101045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766626.484978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766627.125011:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766627.508992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766628.149072:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766628.532960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766628.532997:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766628.533006:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766628.533012:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766628.533021:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3460 00000100:00080000:0.0:1775766628.533033:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766628.533040:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766628.533089:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066436096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766628.533129:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766628.533133:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766628.533135:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766628.533140:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3460 00000100:00080000:0.0:1775766628.533146:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766628.533151:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766628.533165:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766628.533172:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066436096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766628.533202:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066436224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766628.533217:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766628.533221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766628.533223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766628.533230:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3460 00000100:00080000:0.0:1775766628.533235:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766628.533239:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766628.533245:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766628.533252:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766628.533267:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066436352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766628.533277:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766628.533280:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766628.533282:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766628.533290:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3460 00000100:00100000:3.0:1775766628.533291:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766628/real 1775766628] req@ffff9d81901ad180 x1862023066436096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766628.533298:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766628.533309:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766628.533322:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766628.533331:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066436096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766628.533337:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066436480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766628.533372:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766628.533374:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766628.533380:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766628.533380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:0.0:1775766628.533386:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766628.533392:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066436096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766628.533399:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3460 00000100:00080000:0.0:1775766628.533409:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766628.533415:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766628.533419:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766628.533427:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066436608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766628.533432:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766628.533470:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766628.533481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766628/real 1775766628] req@ffff9d81901ae680 x1862023066436224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766628.533490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066436224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766628.533496:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766628.533498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766628.533501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766628.533505:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766628.533515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766628.533522:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766628/real 1775766628] req@ffff9d81901aed80 x1862023066436352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766628.533529:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066436352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766628.533534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766628.533536:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766628.533538:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766628.533541:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766628.533569:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766628.533575:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766628/real 1775766628] req@ffff9d81901afb80 x1862023066436480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766628.533582:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066436480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766628.533589:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766628.533591:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766628.533593:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766628.533603:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766628.533613:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766628.533618:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766628/real 1775766628] req@ffff9d81901af100 x1862023066436608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766628.533626:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066436608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766628.533632:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766628.533634:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766628.533636:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766629.173036:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766629.557025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766630.197028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766630.580945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766631.221080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766631.605015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766632.245160:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766632.628950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766633.269056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766633.653008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766633.653152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766633.653175:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766633.653178:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766633.653185:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3465 00000100:00080000:1.0:1775766633.653195:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766633.653203:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766633.653252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066436736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766633.653279:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766633.653282:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766633.653283:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766633.653286:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3465 00000100:00080000:1.0:1775766633.653289:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766633.653293:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766633.653301:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766633.653311:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066436864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766633.653319:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766633.653324:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766633.653324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066436736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766633.653325:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766633.653333:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3465 00000100:00080000:1.0:1775766633.653335:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766633.653338:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766633.653341:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766633.653351:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066436992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766633.653356:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766633.653359:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766633.653360:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766633.653363:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3465 00000100:00080000:1.0:1775766633.653374:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766633.653376:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766633.653380:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766633.653384:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766633.653390:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066437120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766633.653397:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766633.653399:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766633.653400:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766633.653402:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3465 00000100:00080000:1.0:1775766633.653405:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766633.653407:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766633.653410:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766633.653419:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066437248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766633.653425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766633/real 1775766633] req@ffff9d8190553100 x1862023066436736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766633.653441:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066436736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766633.653449:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766633.653452:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766633.653455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066436736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766633.653471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766633.653504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766633.653513:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766633/real 1775766633] req@ffff9d8190551500 x1862023066436864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766633.653522:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066436864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766633.653528:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766633.653531:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766633.653533:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766633.653538:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766633.653550:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766633.653556:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766633/real 1775766633] req@ffff9d8190551880 x1862023066436992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766633.653565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066436992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766633.653570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766633.653572:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766633.653574:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066436992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766633.653578:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766633.653590:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766633.653596:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766633/real 1775766633] req@ffff9d8190553480 x1862023066437120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766633.653604:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066437120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766633.653611:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766633.653613:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766633.653615:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766633.653619:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766633.653630:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766633.653636:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766633/real 1775766633] req@ffff9d8190550e00 x1862023066437248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766633.653645:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066437248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766633.653651:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766633.653653:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766633.653655:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766634.293071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766634.676951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766635.317051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766635.701006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766636.341086:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766636.724941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766637.365079:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766637.748961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766638.389004:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766638.772940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766638.772993:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766638.772999:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766638.773003:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766638.773012:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3470 00000100:00080000:0.0:1775766638.773023:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766638.773031:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766638.773074:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066437376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766638.773113:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766638.773117:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766638.773118:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766638.773121:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3470 00000100:00080000:0.0:1775766638.773125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766638.773128:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766638.773137:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766638.773150:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066437504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766638.773159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766638.773163:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066437376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766638.773164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766638.773166:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766638.773174:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3470 00000100:00080000:0.0:1775766638.773177:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766638.773179:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766638.773182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766638.773193:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066437632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766638.773199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766638.773202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766638.773204:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766638.773207:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3470 00000100:00080000:0.0:1775766638.773218:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766638.773221:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766638.773226:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766638.773229:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766638.773238:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066437760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766638.773246:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766638.773249:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766638.773250:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766638.773253:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3470 00000100:00080000:0.0:1775766638.773255:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766638.773258:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766638.773262:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766638.773271:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066437888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766638.773286:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766638/real 1775766638] req@ffff9d81901af100 x1862023066437376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766638.773312:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066437376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766638.773325:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766638.773328:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766638.773332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066437376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766638.773353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766638.773374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766638.773385:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766638/real 1775766638] req@ffff9d81901afb80 x1862023066437504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766638.773395:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066437504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766638.773407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766638.773426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766638.773429:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766638.773433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766638.773447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766638.773454:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766638/real 1775766638] req@ffff9d81901aed80 x1862023066437632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766638.773463:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066437632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766638.773470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766638.773472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766638.773475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766638.773479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766638.773492:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766638.773499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766638/real 1775766638] req@ffff9d81901ae680 x1862023066437760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766638.773508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066437760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766638.773515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766638.773517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766638.773520:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766638.773525:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766638.773536:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766638.773543:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766638/real 1775766638] req@ffff9d81901ad180 x1862023066437888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766638.773551:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066437888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766638.773558:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766638.773560:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766638.773562:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066437888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766639.413005:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766639.796955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766640.437067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766640.820974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766641.461033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766641.845014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766642.485034:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766642.868945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766643.509154:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766643.892954:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766643.892962:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766643.892966:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766643.892972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766643.892975:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3475 00000100:00080000:0.0:1775766643.892986:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766643.892992:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766643.893026:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066438016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766643.893061:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766643.893064:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766643.893065:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766643.893068:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3475 00000100:00080000:0.0:1775766643.893070:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766643.893072:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766643.893079:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766643.893088:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066438144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766643.893096:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766643.893100:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766643.893100:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066438016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766643.893101:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766643.893108:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3475 00000100:00080000:0.0:1775766643.893112:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766643.893114:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766643.893117:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766643.893126:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066438272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766643.893133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766643.893135:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766643.893137:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766643.893140:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3475 00000100:00080000:0.0:1775766643.893142:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766643.893144:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766643.893148:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766643.893154:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066438400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766643.893159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766643.893161:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766643.893162:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766643.893165:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3475 00000100:00080000:0.0:1775766643.893168:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766643.893171:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766643.893174:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766643.893178:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766643.893187:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066438528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766643.893227:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766643/real 1775766643] req@ffff9d81901ad180 x1862023066438016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766643.893242:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066438016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766643.893250:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766643.893252:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766643.893255:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066438016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766643.893272:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766643.893288:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766643.893296:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766643/real 1775766643] req@ffff9d81901ae680 x1862023066438144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766643.893303:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066438144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766643.893309:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766643.893311:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766643.893313:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766643.893316:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766643.893326:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766643.893331:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766643/real 1775766643] req@ffff9d81901aed80 x1862023066438272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766643.893338:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066438272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766643.893343:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766643.893345:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766643.893364:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766643.893368:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766643.893380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766643.893386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766643/real 1775766643] req@ffff9d81901afb80 x1862023066438400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766643.893394:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066438400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766643.893401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766643.893403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766643.893406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766643.893409:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766643.893419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766643.893425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766643/real 1775766643] req@ffff9d81901af100 x1862023066438528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766643.893432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066438528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766643.893438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766643.893440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766643.893442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766644.533025:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766644.916950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766645.557043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766645.940969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766646.581026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766646.965040:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766647.605061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766647.988943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766648.629004:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766649.012974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766649.013029:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766649.013035:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766649.013038:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766649.013045:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3480 00000100:00080000:3.0:1775766649.013054:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766649.013059:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766649.013092:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066438656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766649.013128:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766649.013131:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766649.013132:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766649.013135:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3480 00000100:00080000:3.0:1775766649.013138:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766649.013141:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766649.013149:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766649.013174:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066438784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766649.013179:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066438656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766649.013193:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766649.013195:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766649.013196:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766649.013200:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3480 00000100:00080000:3.0:1775766649.013202:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766649.013205:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766649.013209:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766649.013218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066438912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766649.013239:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766649.013242:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766649.013243:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766649.013245:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3480 00000100:00080000:3.0:1775766649.013258:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766649.013261:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766649.013265:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766649.013270:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766649.013284:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066439040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766649.013309:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766649.013311:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766649.013313:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766649.013316:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3480 00000100:00080000:3.0:1775766649.013318:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766649.013322:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766649.013326:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766649.013338:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066439168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766649.013348:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766649/real 1775766649] req@ffff9d8188743800 x1862023066438656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766649.013397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066438656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766649.013410:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766649.013415:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766649.013437:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066438656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766649.013458:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766649.013483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766649.013516:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766649/real 1775766649] req@ffff9d8188741f80 x1862023066438784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766649.013532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066438784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766649.013543:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766649.013547:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766649.013550:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766649.013557:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766649.013651:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766649.013665:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766649/real 1775766649] req@ffff9d8188742300 x1862023066438912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766649.013678:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066438912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766649.013693:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766649.013696:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766649.013698:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066438912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766649.013705:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766649.013725:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766649.013737:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766649/real 1775766649] req@ffff9d8188740700 x1862023066439040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766649.013750:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066439040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766649.013760:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766649.013764:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766649.013767:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766649.013774:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766649.013793:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766649.013805:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766649/real 1775766649] req@ffff9d8188742a00 x1862023066439168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766649.013817:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066439168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766649.013826:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766649.013829:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766649.013833:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766649.653151:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766650.037024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766650.677082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766651.061009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766651.701068:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766652.084957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766652.725129:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766653.108966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766653.749035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766654.132967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766654.133159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766654.133166:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766654.133170:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766654.133178:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3485 00000100:00080000:0.0:1775766654.133187:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766654.133193:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766654.133231:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066439296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766654.133261:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766654.133264:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766654.133265:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766654.133268:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3485 00000100:00080000:0.0:1775766654.133271:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766654.133274:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766654.133294:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766654.133307:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066439296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766654.133313:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066439424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766654.133324:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766654.133329:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766654.133330:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766654.133334:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3485 00000100:00080000:0.0:1775766654.133337:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766654.133340:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766654.133353:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766654.133363:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066439552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766654.133371:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766654.133374:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766654.133375:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766654.133378:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3485 00000800:00000100:3.0:1775766654.133384:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766654.133386:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766654.133390:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766654.133413:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766654.133427:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066439680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766654.133441:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766654.133447:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766654.133450:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775766654.133457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766654/real 1775766654] req@ffff9d81901af100 x1862023066439296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766654.133459:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3485 00000100:00080000:0.0:1775766654.133470:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766654.133481:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766654.133488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066439296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766654.133493:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775766654.133508:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766654.133514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766654.133524:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066439296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766654.133526:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066439808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766654.133595:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766654.133619:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766654.133630:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766654/real 1775766654] req@ffff9d81901afb80 x1862023066439424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766654.133641:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066439424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766654.133647:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766654.133650:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766654.133653:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766654.133657:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766654.133669:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766654.133676:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766654/real 1775766654] req@ffff9d81901aed80 x1862023066439552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766654.133685:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066439552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766654.133691:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766654.133693:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766654.133695:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766654.133699:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766654.133722:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766654.133746:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766654/real 1775766654] req@ffff9d81901ae680 x1862023066439680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766654.133765:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066439680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766654.133774:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766654.133777:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766654.133779:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766654.133785:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766654.133805:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766654.133820:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766654/real 1775766654] req@ffff9d81901ad180 x1862023066439808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766654.133839:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066439808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766654.133853:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766654.133858:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766654.133863:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066439808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766654.773094:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766655.156955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766655.797177:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766656.180944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766656.821070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766657.204997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766657.845028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766658.228999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766658.869001:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766659.252960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766659.253000:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766659.253004:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766659.253008:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766659.253015:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3490 00000100:00080000:2.0:1775766659.253024:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766659.253031:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766659.253063:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066439936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766659.253085:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766659.253091:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766659.253093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766659.253097:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3490 00000100:00080000:2.0:1775766659.253102:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766659.253107:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766659.253117:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766659.253132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066439936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766659.253143:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066440064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766659.253151:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766659.253154:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766659.253155:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766659.253158:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3490 00000100:00080000:2.0:1775766659.253161:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766659.253164:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766659.253167:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766659.253176:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066440192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766659.253182:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766659.253184:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766659.253185:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766659.253198:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3490 00000100:00080000:2.0:1775766659.253201:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766659.253202:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766659.253206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766659.253222:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066440320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766659.253228:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766659.253231:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766659.253232:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766659.253235:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3490 00000100:00080000:2.0:1775766659.253237:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766659.253240:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766659.253246:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766659.253251:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775766659.253257:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066440448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766659.253283:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766659.253298:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766659.253313:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766659.893019:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766660.277014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766660.917051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766661.300950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766661.941038:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766662.324997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775766662.325128:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775766662.325160:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766662.325173:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766662.325200:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766662.325207:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766662.325214:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766662.325220:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775766662.325246:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766659/real 1775766662] req@ffff9d81984a3b80 x1862023066440448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766662.325268:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066440448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766662.325287:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766662.325291:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766662.325294:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440448:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766662.325299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766659/real 1775766662] req@ffff9d81984a3480 x1862023066440064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766662.325307:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066440064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766662.325312:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766662.325315:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766662.325317:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766662.325321:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766659/real 1775766662] req@ffff9d81984a2d80 x1862023066440192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766662.325327:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066440192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766662.325332:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766662.325334:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766662.325336:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766662.325340:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766659/real 1775766662] req@ffff9d81984a0700 x1862023066440320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766662.325348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066440320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766662.325356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766662.325364:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766662.325368:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766662.325376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766659/real 1775766662] req@ffff9d81984a2300 x1862023066439936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766662.325394:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066439936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766662.325401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766662.325405:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766662.325408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066439936:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775766662.965073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766663.348956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766663.989093:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766664.372996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766664.373061:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766664.373069:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766664.373074:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766664.373083:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3495 00000100:00080000:3.0:1775766664.373095:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766664.373115:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766664.373155:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066440576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766664.373192:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766664.373196:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766664.373199:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766664.373203:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3495 00000100:00080000:3.0:1775766664.373206:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766664.373209:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766664.373220:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766664.373241:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066440704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766664.373251:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766664.373254:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766664.373256:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766664.373259:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3495 00000100:00080000:3.0:1775766664.373262:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766664.373265:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766664.373268:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766664.373277:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066440832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766664.373287:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766664.373309:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766664.373311:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766664.373316:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3495 00000100:00080000:3.0:1775766664.373320:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766664.373328:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766664.373331:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066440576:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775766664.373334:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766664.373354:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066440960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766664.373365:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766664.373368:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766664.373370:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766664.373375:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3495 00000100:00080000:3.0:1775766664.373379:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766664.373382:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766664.373387:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766664.373401:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066441088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766664.373415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766664.373545:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766664/real 1775766664] req@ffff9d8188741c00 x1862023066440576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766664.373576:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066440576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766664.373591:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766664.373597:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766664.373603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066440576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766664.373615:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766664.373653:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766664.373674:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766664/real 1775766664] req@ffff9d8188740380 x1862023066440704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766664.373695:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066440704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766664.373709:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766664.373714:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766664.373720:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766664.373729:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766664.373761:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766664.373782:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766664/real 1775766664] req@ffff9d8188740000 x1862023066440832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766664.373803:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066440832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766664.373815:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766664.373820:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766664.373837:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766664.373872:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766664.373959:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766664.373989:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766664/real 1775766664] req@ffff9d8188742a00 x1862023066440960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766664.374009:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066440960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766664.374032:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766664.374037:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766664.374042:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066440960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766664.374051:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766664.374082:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766664.374097:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766664/real 1775766664] req@ffff9d8188740700 x1862023066441088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766664.374113:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066441088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766664.374125:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766664.374129:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766664.374134:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766665.013098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766665.396941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766666.037043:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766666.420974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766667.061050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766667.444949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766668.085026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766668.468993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766669.109038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766669.492951:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766669.492959:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766669.492962:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766669.492971:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3501 00000100:00080000:3.0:1775766669.492981:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766669.492982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766669.492989:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766669.493023:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066441216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766669.493062:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766669.493069:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766669.493072:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766669.493077:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3501 00000100:00080000:3.0:1775766669.493082:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766669.493087:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766669.493097:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766669.493111:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066441344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766669.493123:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766669.493125:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766669.493127:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766669.493130:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3501 00000100:00080000:3.0:1775766669.493133:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766669.493136:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766669.493140:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766669.493144:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066441216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766669.493151:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066441472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766669.493159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766669.493161:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766669.493163:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766669.493165:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3501 00000100:00080000:3.0:1775766669.493167:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766669.493170:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766669.493173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766669.493180:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066441600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766669.493187:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766669.493189:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766669.493191:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766669.493193:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3501 00000100:00080000:3.0:1775766669.493211:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766669.493213:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766669.493216:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:1.0:1775766669.493223:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766669.493224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066441728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766669.493273:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766669/real 1775766669] req@ffff9d8188742300 x1862023066441216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766669.493289:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066441216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766669.493297:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766669.493300:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766669.493303:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066441216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766669.493308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766669.493325:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766669.493334:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766669/real 1775766669] req@ffff9d8188741f80 x1862023066441344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766669.493341:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066441344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766669.493347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766669.493349:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766669.493351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766669.493368:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766669.493407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766669.493415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766669/real 1775766669] req@ffff9d8188743800 x1862023066441472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766669.493423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066441472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766669.493429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766669.493431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766669.493433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766669.493437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766669.493448:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766669.493453:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766669/real 1775766669] req@ffff9d8188742680 x1862023066441600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766669.493460:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066441600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766669.493466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766669.493468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766669.493469:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766669.493473:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766669.493482:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766669.493487:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766669/real 1775766669] req@ffff9d8188741880 x1862023066441728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766669.493495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066441728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766669.493501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766669.493503:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766669.493504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766670.133060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766670.516970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766671.157115:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766671.540986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766672.181063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766672.564957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766673.205009:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766673.588942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766674.229135:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766674.613035:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766674.613052:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766674.613060:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766674.613076:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3506 00000100:00080000:3.0:1775766674.613108:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775766674.613122:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766674.613135:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766674.613203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066441856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766674.613236:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766674.613242:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766674.613244:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766674.613275:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3506 00000100:00080000:3.0:1775766674.613282:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766674.613287:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766674.613299:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766674.613318:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066441984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766674.613352:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066441856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766674.613352:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766674.613363:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766674.613374:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766674.613379:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3506 00000100:00080000:3.0:1775766674.613383:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766674.613387:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766674.613393:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766674.613407:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066442112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766674.613416:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766674.613420:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766674.613421:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766674.613424:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3506 00000100:00080000:3.0:1775766674.613427:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766674.613430:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775766674.613435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766674.613437:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766674.613455:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066442240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766674.613476:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766674.613481:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766674.613484:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766674.613490:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766674/real 1775766674] req@ffff9d8188741180 x1862023066441856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766674.613494:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3506 00000100:00080000:3.0:1775766674.613505:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766674.613512:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766674.613518:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066441856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775766674.613523:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:1.0:1775766674.613536:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766674.613541:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766674.613544:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066442368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766674.613549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066441856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766674.613582:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766674.613606:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766674.613616:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766674/real 1775766674] req@ffff9d8188743100 x1862023066441984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766674.613626:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066441984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766674.613638:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766674.613641:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766674.613644:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066441984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766674.613649:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766674.613662:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766674.613669:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766674/real 1775766674] req@ffff9d8188740e00 x1862023066442112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766674.613678:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066442112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766674.613684:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766674.613687:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766674.613689:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766674.613694:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766674.613706:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766674.613713:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766674/real 1775766674] req@ffff9d8188740a80 x1862023066442240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766674.613723:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066442240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766674.613730:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766674.613732:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766674.613735:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766674.613740:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766674.613752:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766674.613759:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766674/real 1775766674] req@ffff9d8188741500 x1862023066442368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766674.613767:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066442368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766674.613774:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766674.613777:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766674.613779:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766675.253083:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766675.637028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766676.277110:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766676.660978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766677.301122:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766677.684942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766678.325122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766678.709044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766679.349086:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766679.732997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766679.733036:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766679.733042:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766679.733045:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766679.733054:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3511 00000100:00080000:2.0:1775766679.733066:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766679.733073:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766679.733119:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066442496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766679.733147:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766679.733150:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766679.733151:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766679.733154:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3511 00000100:00080000:2.0:1775766679.733156:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766679.733159:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766679.733167:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766679.733177:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066442624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766679.733184:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766679.733187:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766679.733188:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766679.733192:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3511 00000100:00080000:2.0:1775766679.733195:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766679.733197:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766679.733200:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766679.733207:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066442752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766679.733213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766679.733216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766679.733217:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766679.733220:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3511 00000100:00080000:2.0:1775766679.733223:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766679.733225:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766679.733229:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766679.733236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066442880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766679.733243:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766679.733246:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766679.733249:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775766679.733251:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066442496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766679.733255:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3511 00000100:00080000:2.0:1775766679.733266:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766679.733287:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766679.733291:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766679.733300:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066443008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766679.733359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766679.733400:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766679/real 1775766679] req@ffff9d81984a2a00 x1862023066442496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766679.733415:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066442496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766679.733424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766679.733427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766679.733430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066442496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766679.733436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766679.733452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766679.733460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766679/real 1775766679] req@ffff9d81984a2300 x1862023066442624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766679.733468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066442624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766679.733473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766679.733475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766679.733478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766679.733481:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766679.733491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766679.733496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766679/real 1775766679] req@ffff9d81984a0700 x1862023066442752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766679.733503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066442752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766679.733508:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766679.733510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766679.733512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766679.733531:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766679.733542:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766679.733547:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766679/real 1775766679] req@ffff9d81984a2d80 x1862023066442880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766679.733554:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066442880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766679.733562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766679.733564:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766679.733566:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066442880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766679.733569:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766679.733578:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766679.733583:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766679/real 1775766679] req@ffff9d81984a3480 x1862023066443008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766679.733590:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066443008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766679.733596:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766679.733597:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766679.733599:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766680.373325:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766680.757006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766681.397177:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766681.781047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766682.421118:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766682.805042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766683.445099:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766683.829043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766684.469092:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766684.853044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766684.853073:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766684.853081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766684.853086:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766684.853095:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3516 00000100:00080000:1.0:1775766684.853103:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766684.853109:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766684.853144:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066443136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766684.853190:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766684.853194:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766684.853195:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766684.853198:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3516 00000100:00080000:1.0:1775766684.853201:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766684.853204:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766684.853211:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766684.853245:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066443264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766684.853263:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766684.853266:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766684.853269:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766684.853272:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3516 00000100:00080000:1.0:1775766684.853276:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766684.853280:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766684.853284:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766684.853291:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066443392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766684.853298:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766684.853302:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775766684.853303:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066443136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766684.853304:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766684.853312:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3516 00000100:00080000:1.0:1775766684.853314:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766684.853316:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766684.853320:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766684.853332:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066443520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766684.853339:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766684.853342:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766684.853344:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766684.853347:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3516 00000100:00080000:1.0:1775766684.853357:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766684.853360:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766684.853363:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766684.853367:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766684.853375:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066443648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766684.853418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766684/real 1775766684] req@ffff9d8190550e00 x1862023066443136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766684.853433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066443136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766684.853441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766684.853444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766684.853447:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066443136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766684.853453:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766684.853470:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766684.853477:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766684/real 1775766684] req@ffff9d8190553480 x1862023066443264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766684.853485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066443264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766684.853490:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766684.853492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766684.853494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766684.853511:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766684.853522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766684.853527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766684/real 1775766684] req@ffff9d8190551880 x1862023066443392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766684.853534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066443392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766684.853539:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766684.853541:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766684.853543:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766684.853546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766684.853556:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766684.853561:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766684/real 1775766684] req@ffff9d8190551500 x1862023066443520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766684.853568:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066443520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766684.853574:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766684.853575:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766684.853577:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766684.853580:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766684.853590:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766684.853595:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766684/real 1775766684] req@ffff9d8190553100 x1862023066443648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766684.853602:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066443648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766684.853607:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766684.853609:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766684.853611:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766685.493085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766685.876986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766686.517058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766686.900976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766687.541037:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766687.924978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766688.565126:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766688.949029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766689.589098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766689.973011:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766689.973018:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766689.973022:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766689.973024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766689.973031:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3521 00000100:00080000:3.0:1775766689.973044:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766689.973052:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766689.973083:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066443776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766689.973111:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766689.973113:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766689.973115:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766689.973117:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3521 00000100:00080000:3.0:1775766689.973119:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766689.973122:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766689.973136:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766689.973145:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066443904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766689.973152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766689.973155:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766689.973156:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766689.973175:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3521 00000100:00080000:3.0:1775766689.973178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766689.973181:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766689.973186:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066443776:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775766689.973187:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766689.973200:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066444032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766689.973208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766689.973211:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766689.973212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766689.973214:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3521 00000100:00080000:3.0:1775766689.973217:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766689.973219:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766689.973222:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766689.973230:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066444160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766689.973236:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766689.973238:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766689.973240:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766689.973242:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3521 00000800:00000100:0.0:1775766689.973246:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766689.973248:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766689.973252:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766689.973256:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766689.973264:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066444288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766689.973289:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766689/real 1775766689] req@ffff9d8188743480 x1862023066443776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766689.973303:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066443776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766689.973311:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766689.973314:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766689.973317:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066443776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766689.973325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766689.973348:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766689.973360:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766689/real 1775766689] req@ffff9d8188742d80 x1862023066443904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766689.973374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066443904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766689.973383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766689.973386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766689.973389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066443904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766689.973408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766689.973423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766689.973430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766689/real 1775766689] req@ffff9d8188743b80 x1862023066444032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766689.973437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066444032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766689.973443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766689.973445:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766689.973447:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766689.973451:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766689.973461:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766689.973467:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766689/real 1775766689] req@ffff9d8188741500 x1862023066444160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766689.973478:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066444160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766689.973484:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766689.973486:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766689.973488:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766689.973491:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766689.973500:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766689.973505:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766689/real 1775766689] req@ffff9d8188740a80 x1862023066444288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766689.973512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066444288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766689.973518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766689.973519:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766689.973521:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766690.613065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766690.996992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766691.637098:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766692.021025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766692.661062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766693.045001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766693.685158:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766694.068987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766694.709053:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766695.092962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766695.092987:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766695.093003:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766695.093007:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766695.093015:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3526 00000100:00080000:3.0:1775766695.093024:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766695.093029:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766695.093062:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066444416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766695.093089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766695.093092:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766695.093093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766695.093096:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3526 00000100:00080000:3.0:1775766695.093098:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766695.093100:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766695.093118:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766695.093129:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066444544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766695.093141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766695.093144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766695.093147:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766695.093150:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3526 00000100:00080000:3.0:1775766695.093153:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766695.093161:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766695.093165:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066444416:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775766695.093167:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766695.093180:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066444672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766695.093188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766695.093192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766695.093194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766695.093197:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3526 00000100:00080000:3.0:1775766695.093199:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766695.093202:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766695.093205:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766695.093215:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066444800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766695.093222:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766695.093224:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766695.093225:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766695.093228:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3526 00000100:00080000:3.0:1775766695.093238:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766695.093241:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766695.093244:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766695.093255:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066444928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766695.093259:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766695.093350:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766695/real 1775766695] req@ffff9d8188740e00 x1862023066444416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766695.093371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066444416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766695.093381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766695.093385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766695.093388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066444416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766695.093395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766695.093415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766695.093425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766695/real 1775766695] req@ffff9d8188743100 x1862023066444544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766695.093434:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066444544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766695.093440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766695.093443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766695.093446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766695.093477:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766695.093499:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766695.093511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766695/real 1775766695] req@ffff9d8188741180 x1862023066444672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766695.093525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066444672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766695.093541:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766695.093545:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766695.093548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766695.093553:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766695.093570:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766695.093578:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766695/real 1775766695] req@ffff9d8188741880 x1862023066444800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766695.093587:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066444800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766695.093594:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766695.093597:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766695.093599:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766695.093602:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766695.093612:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766695.093618:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766695/real 1775766695] req@ffff9d8188742680 x1862023066444928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766695.093625:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066444928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766695.093630:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766695.093632:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766695.093635:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066444928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766695.733113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766696.116948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766696.757044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766697.140972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766697.781123:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766698.164947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766698.805098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766699.188989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766699.829035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766700.212949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766700.212957:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766700.212966:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766700.212972:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766700.212982:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3531 00000100:00080000:3.0:1775766700.212995:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766700.213002:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766700.213046:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066445056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766700.213086:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766700.213089:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766700.213091:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766700.213094:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3531 00000100:00080000:3.0:1775766700.213097:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766700.213100:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766700.213109:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766700.213120:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066445184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766700.213129:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766700.213139:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066445056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766700.213140:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766700.213144:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766700.213160:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3531 00000100:00080000:3.0:1775766700.213163:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766700.213166:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766700.213171:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766700.213182:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066445312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766700.213190:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766700.213192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766700.213194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766700.213197:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3531 00000100:00080000:3.0:1775766700.213206:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766700.213209:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766700.213214:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766700.213222:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766700.213234:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066445440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766700.213248:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766700.213252:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766700.213254:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766700.213260:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3531 00000100:00080000:3.0:1775766700.213266:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766700.213267:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766700/real 1775766700] req@ffff9d8188743800 x1862023066445056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766700.213280:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766700.213293:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766700.213296:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066445056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766700.213310:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766700.213317:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766700.213317:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066445568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766700.213327:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066445056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766700.213357:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766700.213376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766700.213384:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766700/real 1775766700] req@ffff9d8188741f80 x1862023066445184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766700.213393:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066445184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766700.213399:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766700.213401:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766700.213404:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766700.213408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766700.213419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766700.213424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766700/real 1775766700] req@ffff9d8188742300 x1862023066445312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766700.213431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066445312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766700.213436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766700.213438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766700.213440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766700.213444:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766700.213454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766700.213459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766700/real 1775766700] req@ffff9d8188740700 x1862023066445440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766700.213466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066445440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766700.213472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766700.213474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766700.213476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766700.213480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766700.213489:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766700.213494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766700/real 1775766700] req@ffff9d8188742a00 x1862023066445568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766700.213501:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066445568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766700.213507:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766700.213508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766700.213510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766700.853030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766701.236971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766701.877105:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766702.260937:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766702.901011:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766703.284996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766703.925078:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766704.308950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766704.949113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766705.333065:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766705.333067:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766705.333074:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766705.333079:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766705.333087:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3536 00000100:00080000:1.0:1775766705.333096:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766705.333102:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766705.333137:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066445696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766705.333160:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766705.333163:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766705.333165:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766705.333167:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3536 00000100:00080000:1.0:1775766705.333170:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766705.333173:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766705.333180:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766705.333189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066445824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766705.333196:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766705.333199:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766705.333201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766705.333204:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3536 00000100:00080000:1.0:1775766705.333205:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766705.333207:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766705.333210:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766705.333216:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066445952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766705.333223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766705.333225:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766705.333226:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766705.333228:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3536 00000100:00080000:1.0:1775766705.333231:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766705.333239:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766705.333241:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066445696:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775766705.333243:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766705.333259:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066446080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766705.333269:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766705.333271:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766705.333272:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766705.333274:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3536 00000100:00080000:1.0:1775766705.333276:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766705.333280:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766705.333282:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766705.333289:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066446208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766705.333342:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766705.333418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766705/real 1775766705] req@ffff9d8190553100 x1862023066445696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766705.333436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066445696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766705.333447:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766705.333450:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766705.333453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066445696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766705.333460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766705.333479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766705.333489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766705/real 1775766705] req@ffff9d8190551500 x1862023066445824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766705.333497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066445824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766705.333503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766705.333506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766705.333508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766705.333530:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766705.333542:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766705.333547:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766705/real 1775766705] req@ffff9d8190551880 x1862023066445952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766705.333555:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066445952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766705.333568:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766705.333571:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766705.333573:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066445952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766705.333580:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766705.333593:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766705.333600:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766705/real 1775766705] req@ffff9d8190553480 x1862023066446080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766705.333608:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066446080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766705.333615:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766705.333617:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766705.333620:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766705.333623:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766705.333633:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766705.333639:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766705/real 1775766705] req@ffff9d8190550e00 x1862023066446208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766705.333646:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066446208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766705.333652:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766705.333654:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766705.333655:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766705.973038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766706.356980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766706.997048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766707.381041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766708.021143:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766708.405041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766709.045054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766709.428993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766710.069146:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766710.453075:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766710.453081:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766710.453094:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766710.453099:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766710.453112:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3541 00000100:00080000:3.0:1775766710.453125:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766710.453135:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766710.453182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066446336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766710.453223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766710.453228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766710.453231:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766710.453235:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3541 00000100:00080000:3.0:1775766710.453239:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766710.453243:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766710.453254:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766710.453267:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066446464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766710.453293:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766710.453297:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766710.453299:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766710.453305:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3541 00000100:00080000:3.0:1775766710.453310:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766710.453315:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766710.453323:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766710.453333:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066446336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766710.453342:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066446592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766710.453360:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766710.453364:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766710.453366:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766710.453370:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3541 00000100:00080000:3.0:1775766710.453374:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766710.453378:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766710.453384:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766710.453401:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066446720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766710.453412:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766710.453417:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766710.453419:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766710.453424:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3541 00000100:00080000:3.0:1775766710.453428:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766710.453432:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766710.453439:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766710.453449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766710.453459:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066446848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766710.453534:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766710/real 1775766710] req@ffff9d8188740000 x1862023066446336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766710.453562:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066446336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766710.453577:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766710.453582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766710.453587:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066446336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766710.453598:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766710.453656:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766710.453673:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766710/real 1775766710] req@ffff9d8188740380 x1862023066446464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766710.453690:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066446464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766710.453700:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766710.453704:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766710.453709:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766710.453736:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766710.453758:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766710.453770:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766710/real 1775766710] req@ffff9d8188741c00 x1862023066446592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766710.453784:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066446592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766710.453795:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766710.453798:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766710.453803:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766710.453810:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766710.453839:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766710.453849:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766710/real 1775766710] req@ffff9d8188742a00 x1862023066446720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766710.453863:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066446720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766710.453917:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766710.453923:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766710.453927:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766710.453935:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766710.453991:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766710.454008:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766710/real 1775766710] req@ffff9d8188740700 x1862023066446848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766710.454024:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066446848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766710.454034:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766710.454038:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766710.454042:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066446848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766711.093134:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766711.477017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766712.117158:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766712.500999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766713.141105:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766713.524970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766714.165045:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766714.548963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766715.189174:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766715.572993:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766715.572994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766715.573004:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766715.573009:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766715.573018:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3547 00000100:00080000:3.0:1775766715.573030:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766715.573040:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766715.573094:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066446976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766715.573126:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766715.573130:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766715.573131:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766715.573135:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3547 00000100:00080000:3.0:1775766715.573139:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766715.573143:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766715.573152:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766715.573165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066447104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766715.573175:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766715.573178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766715.573180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766715.573183:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3547 00000100:00080000:3.0:1775766715.573186:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766715.573191:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766715.573193:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066446976:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775766715.573197:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766715.573208:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066447232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766715.573224:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766715.573228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766715.573230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766715.573234:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3547 00000100:00080000:3.0:1775766715.573249:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766715.573252:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766715.573256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766715.573261:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766715.573274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066447360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766715.573284:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766715.573287:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766715.573288:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766715.573292:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3547 00000100:00080000:3.0:1775766715.573296:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766715.573300:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766715.573304:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766715.573310:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766715/real 1775766715] req@ffff9d8188742300 x1862023066446976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766715.573316:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066447488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766715.573390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066446976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766715.573404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766715.573410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766715.573415:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066446976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766715.573423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766715.573448:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766715.573461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766715/real 1775766715] req@ffff9d8188741f80 x1862023066447104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766715.573474:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066447104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766715.573483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766715.573487:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766715.573490:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766715.573515:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766715.573534:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766715.573543:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766715/real 1775766715] req@ffff9d8188743800 x1862023066447232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766715.573556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066447232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766715.573564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766715.573567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766715.573570:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766715.573576:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766715.573603:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766715.573612:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766715/real 1775766715] req@ffff9d8188742680 x1862023066447360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766715.573623:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066447360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766715.573633:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766715.573636:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766715.573639:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766715.573644:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766715.573659:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766715.573667:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766715/real 1775766715] req@ffff9d8188741880 x1862023066447488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766715.573692:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066447488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766715.573707:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766715.573712:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766715.573717:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766716.213022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766716.596994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766717.237076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766717.620975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766718.261100:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766718.644959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766719.285067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766719.668943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766720.309049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766720.692980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766720.692993:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766720.692999:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766720.693003:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766720.693011:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3552 00000100:00080000:0.0:1775766720.693019:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766720.693024:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766720.693059:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066447616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766720.693084:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766720.693087:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766720.693088:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766720.693091:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3552 00000100:00080000:0.0:1775766720.693093:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766720.693096:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766720.693103:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766720.693112:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066447744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766720.693119:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766720.693122:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766720.693124:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766720.693127:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3552 00000100:00080000:0.0:1775766720.693129:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766720.693134:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766720.693134:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066447616:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775766720.693141:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766720.693148:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066447872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766720.693155:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766720.693157:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766720.693159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766720.693161:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3552 00000100:00080000:0.0:1775766720.693163:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766720.693165:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766720.693168:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766720.693176:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066448000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766720.693182:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766720.693185:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766720.693186:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766720.693189:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3552 00000800:00000100:1.0:1775766720.693193:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766720.693194:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766720.693198:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766720.693202:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766720.693210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066448128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766720.693238:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766720/real 1775766720] req@ffff9d81901ad180 x1862023066447616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766720.693251:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066447616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766720.693258:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766720.693261:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766720.693264:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066447616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766720.693269:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766720.693283:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766720.693290:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766720/real 1775766720] req@ffff9d81901ae680 x1862023066447744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766720.693298:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066447744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766720.693304:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766720.693306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766720.693308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766720.693324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766720.693335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766720.693340:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766720/real 1775766720] req@ffff9d81901aed80 x1862023066447872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766720.693347:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066447872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766720.693352:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766720.693354:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766720.693356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066447872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766720.693359:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766720.693369:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766720.693374:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766720/real 1775766720] req@ffff9d81901afb80 x1862023066448000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766720.693381:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066448000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766720.693392:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766720.693394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766720.693396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766720.693399:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766720.693408:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766720.693412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766720/real 1775766720] req@ffff9d81901af100 x1862023066448128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766720.693419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066448128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766720.693424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766720.693426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766720.693428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766721.333134:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766721.717001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766722.357089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766722.740988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766723.381172:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766723.764984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766724.405036:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766724.788989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766725.429012:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766725.812970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766725.812972:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766725.812981:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766725.812986:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766725.812997:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3557 00000100:00080000:2.0:1775766725.813008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766725.813016:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766725.813053:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066448256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766725.813083:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766725.813087:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766725.813090:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766725.813094:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3557 00000100:00080000:2.0:1775766725.813098:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766725.813102:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766725.813123:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766725.813138:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066448384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766725.813150:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766725.813155:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766725.813157:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766725.813166:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3557 00000100:00080000:2.0:1775766725.813171:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766725.813176:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766725.813181:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766725.813203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066448512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766725.813204:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066448256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766725.813230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766725.813234:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766725.813236:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766725.813243:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3557 00000100:00080000:2.0:1775766725.813248:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766725.813252:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766725.813258:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766725.813274:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066448640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766725.813286:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766725.813289:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766725.813291:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766725.813297:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3557 00000100:00080000:2.0:1775766725.813303:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766725.813308:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766725.813314:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766725.813327:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066448768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766725.813373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766725.813418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766725.813450:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766725.813493:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766726.453141:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766726.837013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766727.477105:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766727.860962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766728.501075:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766728.884956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775766728.885162:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775766728.885200:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775766728.885216:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766728.885264:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766728.885271:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766728.885276:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766728.885281:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775766728.885388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766725/real 1775766728] req@ffff9d81984a2680 x1862023066448512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766728.885425:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066448512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766728.885437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766728.885441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766728.885445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766728.885452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766725/real 1775766728] req@ffff9d81984a0e00 x1862023066448640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766728.885462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066448640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766728.885469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766728.885472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766728.885474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766728.885479:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766725/real 1775766728] req@ffff9d81984a1f80 x1862023066448768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766728.885487:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066448768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766728.885494:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766728.885497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766728.885499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448768:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766728.885504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766725/real 1775766728] req@ffff9d81984a3b80 x1862023066448256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766728.885514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066448256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766728.885520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766728.885523:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766728.885525:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066448256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766728.885530:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766725/real 1775766728] req@ffff9d81984a0000 x1862023066448384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766728.885538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066448384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766728.885543:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766728.885546:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766728.885548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066448384:192.168.202.105@tcp:38:kworker.0 00010000:00010000:0.0:1775766729.525051:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766729.909073:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766730.549116:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766730.932993:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766730.932996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766730.933008:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766730.933014:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766730.933027:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3562 00000100:00080000:2.0:1775766730.933039:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766730.933058:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766730.933094:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066448896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766730.933140:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766730.933145:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766730.933147:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766730.933152:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3562 00000100:00080000:2.0:1775766730.933155:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766730.933160:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766730.933171:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766730.933194:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066449024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766730.933213:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066448896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766730.933219:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766730.933225:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766730.933227:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766730.933233:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3562 00000100:00080000:2.0:1775766730.933237:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766730.933242:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766730.933248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766730.933261:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066449152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766730.933271:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766730.933280:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766730.933284:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766730.933287:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766730.933293:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3562 00000100:00080000:2.0:1775766730.933299:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766730.933303:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766730.933310:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766730.933311:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766730/real 1775766730] req@ffff9d81984a1180 x1862023066448896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766730.933327:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066449280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766730.933339:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066448896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766730.933340:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766730.933349:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766730.933354:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766730.933362:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775766730.933363:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3562 00000100:00080000:2.0:1775766730.933371:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766730.933372:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775766730.933380:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766730.933382:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066448896:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775766730.933391:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766730.933407:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066449408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766730.933416:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766730.933447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766730.933461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766730/real 1775766730] req@ffff9d81984a1880 x1862023066449024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766730.933472:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066449024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766730.933479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766730.933483:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766730.933485:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766730.933491:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766730.933514:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766730.933523:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766730/real 1775766730] req@ffff9d81984a3800 x1862023066449152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766730.933535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066449152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766730.933542:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766730.933545:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766730.933548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766730.933553:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766730.933574:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766730.933585:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766730/real 1775766730] req@ffff9d81984a3100 x1862023066449280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766730.933597:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066449280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766730.933606:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766730.933609:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766730.933611:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766730.933627:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766730.933647:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766730.933657:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766730/real 1775766730] req@ffff9d81984a1500 x1862023066449408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766730.933669:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066449408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766730.933678:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766730.933680:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766730.933683:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766731.573035:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766731.956961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766732.597060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766732.980967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766733.621034:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766734.004960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766734.644994:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766735.028964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766735.669113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766736.053007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766736.053050:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766736.053060:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766736.053067:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766736.053081:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3567 00000100:00080000:2.0:1775766736.053096:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766736.053135:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766736.053212:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066449536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766736.053246:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766736.053253:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766736.053257:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766736.053266:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3567 00000100:00080000:2.0:1775766736.053274:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766736.053280:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766736.053294:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766736.053312:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066449536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766736.053316:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066449664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766736.053333:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766736.053337:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766736.053351:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766736.053357:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3567 00000100:00080000:2.0:1775766736.053363:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766736.053368:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766736.053374:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766736.053389:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066449792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766736.053395:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766736.053405:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766736.053410:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766736.053413:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766736.053418:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3567 00000100:00080000:2.0:1775766736.053423:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766736.053428:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766736.053435:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766736.053449:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066449920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766736.053464:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766736.053465:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766736/real 1775766736] req@ffff9d81984a0a80 x1862023066449536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766736.053470:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766736.053474:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766736.053497:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3567 00000100:00080000:2.0:1775766736.053504:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766736.053511:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766736.053515:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066449536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775766736.053521:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:1.0:1775766736.053534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1775766736.053539:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066450048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766736.053541:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766736.053549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066449536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766736.053582:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766736.053625:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766736.053662:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766736/real 1775766736] req@ffff9d81984a1500 x1862023066449664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766736.053710:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066449664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766736.053719:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766736.053723:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766736.053727:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766736.053734:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766736.053758:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766736.053770:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766736/real 1775766736] req@ffff9d81984a3100 x1862023066449792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766736.053782:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066449792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766736.053790:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766736.053793:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766736.053798:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766736.053804:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766736.053820:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766736.053829:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766736/real 1775766736] req@ffff9d81984a3800 x1862023066449920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766736.053840:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066449920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766736.053849:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766736.053852:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766736.053855:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066449920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766736.053860:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766736.053902:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766736.053914:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766736/real 1775766736] req@ffff9d81984a1880 x1862023066450048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766736.053927:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066450048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766736.053946:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766736.053949:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766736.053953:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766736.693032:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766737.076952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766737.717025:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766738.100998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766738.741182:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766739.125027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766739.765065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766740.149020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766740.789127:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766741.173039:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766741.173048:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766741.173052:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766741.173060:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3572 00000100:00100000:2.0:1775766741.173069:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766741.173070:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766741.173078:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766741.173112:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066450176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766741.173142:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766741.173145:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766741.173147:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766741.173149:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3572 00000100:00080000:3.0:1775766741.173151:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766741.173154:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766741.173162:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766741.173174:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066450304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766741.173182:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066450176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766741.173189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766741.173196:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766741.173198:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766741.173204:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3572 00000100:00080000:3.0:1775766741.173209:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766741.173214:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766741.173219:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766741.173233:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066450432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766741.173244:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766741.173248:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766741.173250:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766741.173254:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3572 00000100:00080000:3.0:1775766741.173266:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766741.173269:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766741.173274:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766741.173282:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766741.173292:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066450560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766741.173301:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766741.173310:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766741.173312:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766741.173315:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3572 00000100:00080000:3.0:1775766741.173320:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766741.173325:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766741.173331:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766741.173344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766741/real 1775766741] req@ffff9d8188741180 x1862023066450176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766741.173345:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066450688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766741.173415:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066450176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766741.173429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766741.173435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766741.173442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066450176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766741.173474:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766741.173499:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766741.173513:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766741/real 1775766741] req@ffff9d8188743100 x1862023066450304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766741.173527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066450304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766741.173537:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766741.173542:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766741.173545:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766741.173552:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766741.173572:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766741.173580:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766741/real 1775766741] req@ffff9d8188740e00 x1862023066450432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766741.173588:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066450432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766741.173594:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766741.173597:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766741.173599:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766741.173603:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766741.173615:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766741.173621:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766741/real 1775766741] req@ffff9d8188740a80 x1862023066450560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766741.173653:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066450560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766741.173661:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766741.173663:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766741.173665:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766741.173669:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766741.173684:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766741.173691:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766741/real 1775766741] req@ffff9d8188741500 x1862023066450688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766741.173699:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066450688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766741.173704:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766741.173706:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766741.173708:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766741.813077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766742.197034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766742.837152:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766743.220995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766743.861048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766744.245017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766744.885075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766745.269026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766745.909089:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766746.293041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766746.293082:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766746.293098:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766746.293107:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766746.293121:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3577 00000100:00080000:1.0:1775766746.293136:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766746.293145:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766746.293189:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066450816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766746.293230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766746.293234:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766746.293237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766746.293241:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3577 00000100:00080000:1.0:1775766746.293245:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766746.293249:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766746.293260:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766746.293272:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066450944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766746.293284:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766746.293288:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766746.293302:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766746.293308:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3577 00000100:00080000:1.0:1775766746.293311:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766746.293315:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766746.293323:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766746.293325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066450816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766746.293339:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066451072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766746.293366:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766746.293370:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766746.293372:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766746.293378:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3577 00000100:00080000:1.0:1775766746.293396:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766746.293400:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766746.293405:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766746.293406:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766746.293424:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066451200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766746.293436:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766746.293439:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766746.293441:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766746.293446:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3577 00000100:00080000:1.0:1775766746.293450:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766746.293454:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766746.293460:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766746.293482:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066451328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766746.293510:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766746/real 1775766746] req@ffff9d8190550e00 x1862023066450816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766746.293545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066450816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766746.293564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766746.293599:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766746.293618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066450816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766746.293630:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766746.293661:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766746.293678:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766746/real 1775766746] req@ffff9d8190553480 x1862023066450944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766746.293692:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066450944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766746.293701:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766746.293705:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766746.293708:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066450944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766746.293751:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766746.293769:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766746.293779:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766746/real 1775766746] req@ffff9d8190551880 x1862023066451072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766746.293791:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066451072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766746.293799:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766746.293802:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766746.293805:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766746.293810:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766746.293825:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766746.293833:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766746/real 1775766746] req@ffff9d8190551500 x1862023066451200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766746.293844:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066451200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766746.293854:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766746.293858:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766746.293861:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766746.293866:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766746.293880:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766746.293888:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766746/real 1775766746] req@ffff9d8190553100 x1862023066451328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766746.293939:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066451328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766746.293948:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766746.293952:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766746.293955:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766746.933103:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766747.316973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766747.957011:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766748.341009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766748.981041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766749.364961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766750.005038:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766750.388962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766751.029175:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766751.412964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766751.413103:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766751.413116:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766751.413122:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766751.413133:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3582 00000100:00080000:3.0:1775766751.413147:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766751.413155:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766751.413203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066451456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766751.413254:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766751.413262:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766751.413266:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766751.413276:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3582 00000100:00080000:3.0:1775766751.413284:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766751.413293:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766751.413308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066451456:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775766751.413311:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766751.413347:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066451584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766751.413373:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766751.413377:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766751.413380:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766751.413386:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3582 00000100:00080000:3.0:1775766751.413396:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766751.413401:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766751.413409:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766751.413411:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766751.413430:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066451712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766751.413444:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766751.413448:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766751.413451:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766751.413459:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3582 00000100:00080000:3.0:1775766751.413465:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766751.413470:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766751.413477:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766751.413483:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766751/real 1775766751] req@ffff9d8188743b80 x1862023066451456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766751.413498:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066451840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766751.413511:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766751.413542:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066451456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766751.413546:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766751.413555:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766751.413572:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3582 00000100:00080000:3.0:1775766751.413585:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766751.413599:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775766751.413600:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766751.413618:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:3.0:1775766751.413618:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766751.413629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066451456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766751.413645:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066451968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766751.413663:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766751.413695:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766751.413707:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766751/real 1775766751] req@ffff9d8188742d80 x1862023066451584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766751.413716:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066451584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766751.413722:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766751.413725:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766751.413727:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766751.413731:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766751.413743:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766751.413749:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766751/real 1775766751] req@ffff9d8188743480 x1862023066451712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766751.413756:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066451712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766751.413762:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766751.413764:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766751.413766:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766751.413771:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766751.413784:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766751.413789:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766751/real 1775766751] req@ffff9d8188741500 x1862023066451840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766751.413796:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066451840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766751.413804:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766751.413806:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766751.413808:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766751.413818:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766751.413829:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766751.413835:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766751/real 1775766751] req@ffff9d8188740a80 x1862023066451968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766751.413843:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066451968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766751.413850:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766751.413851:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766751.413853:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066451968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766752.053187:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766752.436992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766753.077032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766753.460937:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766754.101116:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766754.485014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766755.125068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766755.508988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766756.149168:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766756.532986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766756.533014:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766756.533023:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766756.533027:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766756.533034:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3588 00000100:00080000:0.0:1775766756.533043:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766756.533049:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766756.533083:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066452096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766756.533109:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766756.533112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766756.533113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766756.533120:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3588 00000100:00080000:0.0:1775766756.533123:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766756.533125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766756.533133:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766756.533141:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066452224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766756.533148:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766756.533151:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766756.533152:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766756.533155:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3588 00000100:00080000:0.0:1775766756.533157:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766756.533162:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766756.533162:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066452096:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775766756.533169:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766756.533175:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066452352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766756.533183:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766756.533185:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766756.533186:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766756.533189:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3588 00000100:00080000:0.0:1775766756.533191:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766756.533193:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766756.533196:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766756.533203:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066452480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766756.533209:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766756.533212:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766756.533213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766756.533215:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3588 00000100:00080000:0.0:1775766756.533217:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766756.533222:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766756.533224:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766756.533228:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766756.533236:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066452608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766756.533265:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766756/real 1775766756] req@ffff9d81901af100 x1862023066452096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766756.533284:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066452096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766756.533291:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766756.533294:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766756.533297:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066452096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766756.533302:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766756.533316:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766756.533324:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766756/real 1775766756] req@ffff9d81901afb80 x1862023066452224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766756.533332:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066452224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766756.533337:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766756.533339:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766756.533341:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766756.533358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766756.533367:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766756.533373:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766756/real 1775766756] req@ffff9d81901aed80 x1862023066452352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766756.533380:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066452352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766756.533389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766756.533391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766756.533393:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766756.533397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766756.533411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766756.533420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766756/real 1775766756] req@ffff9d81901ae680 x1862023066452480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766756.533432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066452480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766756.533442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766756.533446:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766756.533449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766756.533456:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766756.533492:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766756.533500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766756/real 1775766756] req@ffff9d81901ad180 x1862023066452608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766756.533508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066452608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766756.533517:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766756.533520:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766756.533523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766757.173088:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766757.557092:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766758.197142:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766758.581017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766759.221084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766759.604997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766760.245128:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766760.629028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766761.269032:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766761.652940:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766761.652944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766761.652950:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766761.652954:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766761.652962:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3593 00000100:00080000:0.0:1775766761.652972:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766761.652980:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766761.653016:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066452736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766761.653042:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766761.653045:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766761.653047:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766761.653051:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3593 00000100:00080000:0.0:1775766761.653055:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766761.653059:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766761.653068:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766761.653082:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066452864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766761.653085:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066452736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766761.653092:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766761.653095:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766761.653097:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766761.653101:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3593 00000100:00080000:0.0:1775766761.653104:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766761.653107:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766761.653112:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766761.653121:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066452992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766761.653129:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766761.653132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766761.653134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766761.653137:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3593 00000100:00080000:0.0:1775766761.653140:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775766761.653143:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766761.653144:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766761.653148:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766761.653156:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066453120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766761.653164:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766761.653167:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766761.653169:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766761.653172:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3593 00000100:00080000:0.0:1775766761.653175:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766761.653178:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766761.653182:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766761.653185:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766761/real 1775766761] req@ffff9d81901ad180 x1862023066452736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766761.653191:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066453248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766761.653202:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066452736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766761.653213:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766761.653217:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766761.653221:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066452736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766761.653242:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766761.653260:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766761.653269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766761/real 1775766761] req@ffff9d81901ae680 x1862023066452864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766761.653277:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066452864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766761.653282:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766761.653284:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766761.653286:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766761.653290:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766761.653300:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766761.653307:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766761/real 1775766761] req@ffff9d81901aed80 x1862023066452992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766761.653314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066452992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766761.653319:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766761.653321:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766761.653322:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066452992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766761.653326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766761.653336:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766761.653342:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766761/real 1775766761] req@ffff9d81901afb80 x1862023066453120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766761.653349:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066453120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766761.653354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766761.653356:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766761.653358:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766761.653371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766761.653382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766761.653388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766761/real 1775766761] req@ffff9d81901af100 x1862023066453248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766761.653395:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066453248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766761.653400:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766761.653406:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766761.653409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766762.293077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766762.676980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766763.317041:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766763.700968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766764.341086:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766764.724948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766765.364990:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766765.749061:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766766.389129:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766766.772986:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766766.772987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766766.773001:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766766.773005:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766766.773015:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3598 00000100:00080000:0.0:1775766766.773026:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766766.773032:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766766.773067:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066453376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766766.773109:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766766.773111:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766766.773113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766766.773116:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3598 00000100:00080000:0.0:1775766766.773118:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766766.773121:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766766.773128:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766766.773136:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066453504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766766.773143:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766766.773145:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766766.773146:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766766.773149:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3598 00000100:00080000:0.0:1775766766.773151:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766766.773153:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766766.773156:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766766.773163:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066453632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766766.773170:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766766.773172:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766766.773173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766766.773176:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3598 00000100:00080000:0.0:1775766766.773178:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766766.773180:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766766.773184:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766766.773193:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066453760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766766.773193:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066453376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766766.773206:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766766.773208:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766766.773209:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766766.773212:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3598 00000100:00080000:0.0:1775766766.773214:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766766.773216:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766766.773219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766766.773228:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066453888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766766.773255:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766766.773293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766766/real 1775766766] req@ffff9d81901af100 x1862023066453376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766766.773307:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066453376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766766.773315:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766766.773318:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766766.773321:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066453376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766766.773326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766766.773339:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766766.773347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766766/real 1775766766] req@ffff9d81901afb80 x1862023066453504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766766.773355:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066453504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766766.773392:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766766.773395:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766766.773397:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766766.773401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766766.773415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766766.773421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766766/real 1775766766] req@ffff9d81901aed80 x1862023066453632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766766.773440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066453632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766766.773445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766766.773447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766766.773449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766766.773466:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766766.773476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766766.773482:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766766/real 1775766766] req@ffff9d81901ae680 x1862023066453760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766766.773489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066453760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766766.773495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766766.773496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766766.773498:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766766.773501:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766766.773510:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766766.773516:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766766/real 1775766766] req@ffff9d81901ad180 x1862023066453888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766766.773523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066453888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766766.773528:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766766.773530:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766766.773532:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066453888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766767.413029:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766767.796997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766768.437091:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766768.820944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766769.461153:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766769.845013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766770.485067:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766770.868937:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766771.509136:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775766771.892990:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766771.892999:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766771.893004:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775766771.893004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766771.893015:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3603 00000100:00080000:0.0:1775766771.893029:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766771.893038:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766771.893075:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066454016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766771.893125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766771.893130:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766771.893132:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766771.893137:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3603 00000100:00080000:0.0:1775766771.893143:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766771.893149:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766771.893162:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766771.893190:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066454016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766771.893197:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066454144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766771.893216:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766771.893220:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766771.893222:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766771.893244:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3603 00000100:00080000:0.0:1775766771.893246:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766771.893249:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766771.893254:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766771.893263:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066454272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766771.893271:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766771.893273:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766771.893275:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766771.893277:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3603 00000800:00000100:1.0:1775766771.893283:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766771.893291:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766771.893296:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766771.893300:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766771.893327:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066454400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766771.893334:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766771.893337:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766771.893339:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766771.893343:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3603 00000100:00080000:0.0:1775766771.893347:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766771.893350:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766771.893354:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766771.893359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766771/real 1775766771] req@ffff9d81901ad180 x1862023066454016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766771.893366:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066454528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766771.893396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066454016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766771.893406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766771.893410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766771.893414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066454016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766771.893433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766771.893452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766771.893461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766771/real 1775766771] req@ffff9d81901ae680 x1862023066454144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766771.893471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066454144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766771.893482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766771.893485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766771.893487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766771.893491:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766771.893503:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766771.893518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766771/real 1775766771] req@ffff9d81901aed80 x1862023066454272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766771.893526:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066454272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766771.893532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766771.893533:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766771.893535:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766771.893540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766771.893551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766771.893558:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766771/real 1775766771] req@ffff9d81901afb80 x1862023066454400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766771.893565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066454400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766771.893572:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766771.893574:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766771.893577:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766771.893581:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766771.893615:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766771.893625:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766771/real 1775766771] req@ffff9d81901af100 x1862023066454528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766771.893639:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066454528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766771.893649:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766771.893653:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766771.893656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766772.533204:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766772.917108:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766773.557114:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766773.940980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766774.581137:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766774.965027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766775.605078:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766775.988966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766776.629043:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766777.013039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766777.013074:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766777.013083:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766777.013087:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766777.013095:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3608 00000100:00080000:1.0:1775766777.013104:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766777.013109:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766777.013143:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066454656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766777.013169:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766777.013171:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766777.013173:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766777.013175:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3608 00000100:00080000:1.0:1775766777.013177:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766777.013181:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766777.013189:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766777.013198:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066454784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766777.013205:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066454656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766777.013208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766777.013215:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766777.013216:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766777.013220:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3608 00000100:00080000:1.0:1775766777.013222:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766777.013224:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766777.013227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766777.013235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066454912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766777.013241:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766777.013244:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766777.013245:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766777.013248:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3608 00000100:00080000:1.0:1775766777.013275:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766777.013279:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766777.013285:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766777.013287:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766777.013301:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066455040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766777.013312:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766777.013316:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766777.013318:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766777.013321:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3608 00000100:00080000:1.0:1775766777.013326:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766777.013330:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766777.013335:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766777.013344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766777/real 1775766777] req@ffff9d8190553100 x1862023066454656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766777.013349:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066455168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766777.013371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066454656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766777.013380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766777.013383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766777.013387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066454656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766777.013405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766777.013422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766777.013430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766777/real 1775766777] req@ffff9d8190551500 x1862023066454784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766777.013439:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066454784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766777.013444:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766777.013446:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766777.013449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766777.013452:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766777.013462:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766777.013467:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766777/real 1775766777] req@ffff9d8190551880 x1862023066454912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766777.013475:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066454912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766777.013480:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766777.013482:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766777.013483:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066454912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766777.013486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766777.013496:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766777.013502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766777/real 1775766777] req@ffff9d8190553480 x1862023066455040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766777.013509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066455040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766777.013514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766777.013516:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766777.013518:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766777.013521:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766777.013530:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766777.013535:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766777/real 1775766777] req@ffff9d8190550e00 x1862023066455168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766777.013541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066455168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766777.013551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766777.013553:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766777.013555:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766777.653037:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766778.037016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766778.677041:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766779.061072:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766779.701061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766780.085035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766780.725070:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766781.109031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766781.749065:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766782.132978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766782.133033:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766782.133043:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766782.133047:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766782.133056:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3613 00000100:00080000:0.0:1775766782.133067:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766782.133075:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766782.133112:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066455296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766782.133156:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766782.133160:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766782.133162:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766782.133167:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3613 00000100:00080000:0.0:1775766782.133171:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766782.133175:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766782.133184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766782.133212:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066455424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766782.133229:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766782.133233:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766782.133235:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766782.133241:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3613 00000100:00080000:0.0:1775766782.133246:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766782.133254:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066455296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766782.133255:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766782.133270:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766782.133287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066455552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766782.133300:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766782.133304:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766782.133305:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766782.133309:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3613 00000100:00080000:0.0:1775766782.133313:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766782.133317:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766782.133323:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766782.133333:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066455680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766782.133343:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766782.133346:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766782.133348:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766782.133352:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3613 00000800:00000100:1.0:1775766782.133357:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766782.133364:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766782.133371:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766782.133380:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766782.133395:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066455808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766782.133435:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766782/real 1775766782] req@ffff9d81901af100 x1862023066455296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766782.133455:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066455296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766782.133465:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766782.133469:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766782.133473:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066455296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766782.133481:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766782.133500:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766782.133527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766782/real 1775766782] req@ffff9d81901afb80 x1862023066455424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766782.133538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066455424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766782.133546:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766782.133549:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766782.133552:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766782.133596:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766782.133614:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766782.133624:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766782/real 1775766782] req@ffff9d81901aed80 x1862023066455552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766782.133642:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066455552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766782.133655:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766782.133660:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766782.133665:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766782.133675:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766782.133707:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766782.133723:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766782/real 1775766782] req@ffff9d81901ae680 x1862023066455680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766782.133742:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066455680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766782.133752:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766782.133767:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766782.133772:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766782.133778:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766782.133801:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766782.133811:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766782/real 1775766782] req@ffff9d81901ad180 x1862023066455808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766782.133822:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066455808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766782.133847:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766782.133849:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766782.133852:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066455808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766782.773039:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766783.157042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766783.797100:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766784.180991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766784.821047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766785.204996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766785.845096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766786.228945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766786.869110:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766787.252959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775766787.253034:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766787.253042:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766787.253048:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766787.253061:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3618 00000100:00080000:2.0:1775766787.253074:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766787.253101:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766787.253187:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066455936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766787.253218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766787.253221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766787.253223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766787.253227:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3618 00000100:00080000:2.0:1775766787.253230:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766787.253236:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766787.253247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766787.253257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066455936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775766787.253262:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066456064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766787.253273:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766787.253275:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766787.253277:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766787.253279:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3618 00000100:00080000:2.0:1775766787.253283:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766787.253285:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766787.253289:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766787.253298:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066456192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766787.253306:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766787.253309:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766787.253311:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766787.253314:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3618 00000100:00080000:2.0:1775766787.253326:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766787.253329:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766787.253332:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775766787.253340:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766787.253345:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066456320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766787.253358:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766787.253363:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766787.253365:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766787.253369:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3618 00000100:00080000:2.0:1775766787.253374:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766787.253380:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766787.253387:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766787.253398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766787/real 1775766787] req@ffff9d81984a1180 x1862023066455936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766787.253405:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066456448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766787.253420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066455936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766787.253431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766787.253436:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766787.253441:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066455936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766787.253482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766787.253510:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766787.253526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766787/real 1775766787] req@ffff9d81984a0000 x1862023066456064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766787.253543:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066456064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766787.253555:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766787.253559:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766787.253563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766787.253569:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766787.253588:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766787.253597:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766787/real 1775766787] req@ffff9d81984a3b80 x1862023066456192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766787.253607:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066456192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766787.253614:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766787.253617:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766787.253619:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766787.253624:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766787.253637:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766787.253643:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766787/real 1775766787] req@ffff9d81984a1f80 x1862023066456320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766787.253651:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066456320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766787.253660:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766787.253662:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766787.253665:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766787.253669:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766787.253680:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766787.253686:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766787/real 1775766787] req@ffff9d81984a0e00 x1862023066456448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766787.253694:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066456448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766787.253702:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766787.253704:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766787.253706:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766787.893083:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766788.277023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766788.917092:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766789.301008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766789.941134:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766790.325166:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766790.965132:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766791.348985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766791.989149:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766792.373009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766792.373018:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766792.373029:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766792.373036:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766792.373049:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3623 00000100:00080000:1.0:1775766792.373063:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766792.373084:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766792.373181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066456576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766792.373221:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766792.373225:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766792.373227:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766792.373230:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3623 00000100:00080000:1.0:1775766792.373234:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766792.373238:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766792.373250:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766792.373263:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066456704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766792.373274:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766792.373277:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766792.373279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766792.373285:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066456576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766792.373303:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3623 00000100:00080000:1.0:1775766792.373309:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766792.373314:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766792.373318:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766792.373328:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066456832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766792.373335:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766792.373339:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766792.373340:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766792.373343:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3623 00000100:00080000:1.0:1775766792.373346:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766792.373348:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766792.373353:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766792.373362:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066456960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766792.373368:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766792.373371:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766792.373372:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766792.373374:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3623 00000100:00080000:1.0:1775766792.373380:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766792.373383:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766792.373385:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766792.373394:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066457088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766792.373452:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766792.373484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766792.373500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766792.373516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766793.013121:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766793.397000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766794.037119:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766794.421037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766795.061202:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766795.445049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775766795.445302:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775766795.445353:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775766795.445393:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766795.445455:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766795.445483:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766795.445511:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766795.445536:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766792/real 1775766795] req@ffff9d8190553480 x1862023066456704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1775766795.445537:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766795.445608:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066456704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766795.445629:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766795.445638:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766795.445644:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766795.445656:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766792/real 1775766795] req@ffff9d8190551880 x1862023066456832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766795.445672:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066456832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766795.445684:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766795.445692:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766795.445698:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766795.445711:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766792/real 1775766795] req@ffff9d8190551500 x1862023066456960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766795.445735:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066456960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766795.445766:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766795.445774:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766795.445781:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066456960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766795.445796:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766792/real 1775766795] req@ffff9d8190553100 x1862023066457088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766795.445819:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066457088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766795.445830:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766795.445835:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766795.445840:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457088:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766795.445854:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766792/real 1775766795] req@ffff9d8190550e00 x1862023066456576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766795.445919:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066456576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766795.445935:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766795.445941:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766795.445946:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066456576:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775766796.085155:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766796.469179:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766797.109107:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766797.493010:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766797.493025:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766797.493032:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766797.493072:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3629 00000100:00080000:3.0:1775766797.493088:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766797.493100:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775766797.493152:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775766797.493174:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066457216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766797.493217:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766797.493223:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766797.493225:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766797.493230:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3629 00000100:00080000:3.0:1775766797.493244:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766797.493248:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766797.493263:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766797.493279:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066457344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766797.493295:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766797.493300:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766797.493302:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766797.493313:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3629 00000100:00100000:0.0:1775766797.493317:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066457216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766797.493318:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766797.493327:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766797.493341:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766797.493365:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066457472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766797.493385:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766797.493392:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766797.493396:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766797.493404:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3629 00000100:00080000:3.0:1775766797.493413:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766797.493420:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766797.493431:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766797.493458:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066457600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766797.493482:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766797.493488:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766797.493492:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766797.493500:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3629 00000800:00000100:0.0:1775766797.493508:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766797.493511:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766797.493524:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766797.493535:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766797.493561:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066457728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766797.493628:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766797/real 1775766797] req@ffff9d8188740e00 x1862023066457216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766797.493705:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066457216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766797.493728:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766797.493736:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766797.493745:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066457216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766797.493791:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766797.493857:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766797.493957:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766797/real 1775766797] req@ffff9d8188743100 x1862023066457344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766797.493987:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066457344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766797.494007:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766797.494014:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766797.494021:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766797.494063:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766797.494110:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766797.494134:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766797/real 1775766797] req@ffff9d8188741180 x1862023066457472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766797.494159:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066457472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766797.494177:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766797.494184:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766797.494191:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766797.494217:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766797.494260:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766797.494283:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766797/real 1775766797] req@ffff9d8188741880 x1862023066457600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766797.494306:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066457600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766797.494325:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766797.494332:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766797.494338:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766797.494350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766797.494389:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766797.494410:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766797/real 1775766797] req@ffff9d8188742680 x1862023066457728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766797.494432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066457728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766797.494448:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766797.494454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766797.494461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766798.133210:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766798.517288:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766799.157252:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766799.541104:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766800.181280:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766800.565137:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766801.205204:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766801.589092:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766802.229097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766802.612970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766802.613027:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766802.613035:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766802.613052:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766802.613060:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3634 00000100:00080000:3.0:1775766802.613069:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766802.613074:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766802.613107:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066457856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766802.613144:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766802.613146:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766802.613148:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766802.613150:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3634 00000100:00080000:3.0:1775766802.613152:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766802.613154:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766802.613161:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766802.613170:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066457984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766802.613177:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766802.613180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766802.613181:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766802.613184:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3634 00000100:00080000:3.0:1775766802.613186:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766802.613188:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766802.613192:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766802.613193:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066457856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766802.613202:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066458112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766802.613214:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766802.613216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766802.613217:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766802.613220:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3634 00000100:00080000:3.0:1775766802.613222:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766802.613225:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766802.613228:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766802.613235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066458240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766802.613241:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766802.613243:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766802.613244:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766802.613246:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3634 00000800:00000100:0.0:1775766802.613251:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766802.613252:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766802.613257:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766802.613260:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766802.613268:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066458368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766802.613295:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766802/real 1775766802] req@ffff9d8188743800 x1862023066457856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766802.613310:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066457856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766802.613323:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766802.613326:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766802.613329:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066457856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766802.613333:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766802.613347:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766802.613354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766802/real 1775766802] req@ffff9d8188741f80 x1862023066457984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766802.613361:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066457984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766802.613367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766802.613369:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766802.613371:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066457984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766802.613387:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766802.613396:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766802.613402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766802/real 1775766802] req@ffff9d8188742300 x1862023066458112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766802.613409:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066458112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766802.613414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766802.613416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766802.613418:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766802.613422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766802.613432:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766802.613437:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766802/real 1775766802] req@ffff9d8188740700 x1862023066458240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766802.613444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066458240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766802.613450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766802.613451:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766802.613453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766802.613457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766802.613476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766802.613481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766802/real 1775766802] req@ffff9d8188742a00 x1862023066458368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766802.613488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066458368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766802.613493:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766802.613495:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766802.613496:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766803.253121:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766803.637052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766804.277176:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766804.661133:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766805.301135:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766805.684998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766806.325090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766806.709003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766807.349087:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766807.733045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766807.733072:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766807.733084:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766807.733092:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766807.733106:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3639 00000100:00080000:1.0:1775766807.733122:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766807.733132:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766807.733184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066458496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766807.733232:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766807.733239:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766807.733242:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766807.733249:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3639 00000100:00080000:1.0:1775766807.733257:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766807.733264:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766807.733290:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766807.733301:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066458496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766807.733310:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066458624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766807.733328:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766807.733333:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766807.733336:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766807.733342:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3639 00000100:00080000:1.0:1775766807.733367:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766807.733374:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766807.733380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775766807.733381:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766807.733399:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066458752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766807.733415:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766807.733420:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766807.733424:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766807.733430:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3639 00000100:00080000:1.0:1775766807.733435:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766807.733441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766807/real 1775766807] req@ffff9d8190550e00 x1862023066458496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766807.733441:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766807.733449:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766807.733472:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066458880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766807.733480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066458496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766807.733499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766807.733500:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766807.733507:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:1.0:1775766807.733507:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766807.733510:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766807.733514:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066458496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766807.733516:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3639 00000100:00080000:1.0:1775766807.733522:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766807.733527:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766807.733533:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766807.733545:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766807.733547:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066459008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766807.733581:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766807.733603:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766807/real 1775766807] req@ffff9d8190553100 x1862023066458624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766807.733637:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066458624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766807.733648:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766807.733652:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766807.733656:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766807.733664:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766807.733701:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766807.733714:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766807/real 1775766807] req@ffff9d8190551500 x1862023066458752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766807.733728:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066458752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766807.733737:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766807.733741:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766807.733745:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766807.733750:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766807.733773:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766807.733782:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766807/real 1775766807] req@ffff9d8190551880 x1862023066458880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766807.733795:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066458880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766807.733805:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766807.733808:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766807.733811:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066458880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766807.733864:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766807.733985:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766807.734001:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766807/real 1775766807] req@ffff9d8190553480 x1862023066459008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766807.734015:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066459008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766807.734033:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766807.734056:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766807.734061:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766808.373038:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766808.756981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766809.397038:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766809.781009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766810.421122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766810.804986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766811.445047:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766811.829063:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766812.469034:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766812.853017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766812.853050:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766812.853058:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766812.853062:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766812.853072:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3644 00000100:00080000:0.0:1775766812.853083:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766812.853090:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766812.853126:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066459136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766812.853155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766812.853159:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766812.853161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766812.853164:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3644 00000100:00080000:0.0:1775766812.853167:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766812.853171:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766812.853180:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766812.853192:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066459264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766812.853203:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066459136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766812.853206:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766812.853213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766812.853215:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766812.853220:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3644 00000100:00080000:0.0:1775766812.853223:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766812.853226:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766812.853230:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766812.853248:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066459392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766812.853261:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766812.853266:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766812.853268:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775766812.853273:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766812.853280:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3644 00000100:00080000:0.0:1775766812.853287:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766812.853293:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766812.853302:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766812.853321:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066459520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766812.853332:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766812.853332:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766812/real 1775766812] req@ffff9d81901ad180 x1862023066459136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766812.853340:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766812.853343:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766812.853352:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3644 00000100:00080000:0.0:1775766812.853359:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766812.853364:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766812.853369:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766812.853377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066459136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766812.853381:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066459648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766812.853442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766812.853450:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766812.853456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066459136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766812.853483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766812.853511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766812.853523:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766812/real 1775766812] req@ffff9d81901ae680 x1862023066459264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766812.853537:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066459264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766812.853546:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766812.853549:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766812.853552:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766812.853558:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766812.853573:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766812.853582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766812/real 1775766812] req@ffff9d81901aed80 x1862023066459392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766812.853593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066459392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766812.853601:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766812.853604:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766812.853606:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766812.853612:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766812.853627:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766812.853636:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766812/real 1775766812] req@ffff9d81901afb80 x1862023066459520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766812.853647:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066459520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766812.853662:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766812.853666:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766812.853669:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766812.853674:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766812.853688:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766812.853695:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766812/real 1775766812] req@ffff9d81901af100 x1862023066459648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766812.853706:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066459648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766812.853714:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766812.853716:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766812.853719:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766813.493055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766813.876993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766814.517144:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766814.900969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766815.541161:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766815.924992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766816.565054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766816.948934:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766817.589164:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766817.972974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766817.972998:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766817.973009:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766817.973015:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766817.973026:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3649 00000100:00080000:0.0:1775766817.973040:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766817.973048:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766817.973120:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066459776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766817.973157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766817.973162:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766817.973164:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766817.973168:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3649 00000100:00080000:0.0:1775766817.973172:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766817.973176:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766817.973186:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766817.973197:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066459904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766817.973208:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766817.973212:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766817.973213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766817.973217:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3649 00000100:00080000:0.0:1775766817.973220:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766817.973223:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766817.973227:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766817.973239:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066460032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766817.973255:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766817.973282:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766817.973285:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766817.973293:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3649 00000100:00100000:1.0:1775766817.973294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066459776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766817.973300:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766817.973307:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766817.973314:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766817.973333:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066460160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766817.973348:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766817.973352:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766817.973354:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766817.973357:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3649 00000100:00080000:0.0:1775766817.973372:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766817.973376:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775766817.973383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775766817.973387:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766817.973403:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066460288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766817.973447:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766817/real 1775766817] req@ffff9d81901af100 x1862023066459776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766817.973467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066459776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766817.973477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766817.973481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766817.973485:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066459776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766817.973506:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766817.973537:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766817.973555:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766817/real 1775766817] req@ffff9d81901afb80 x1862023066459904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766817.973574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066459904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766817.973587:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766817.973592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766817.973596:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066459904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766817.973605:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766817.973629:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766817.973639:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766817/real 1775766817] req@ffff9d81901aed80 x1862023066460032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766817.973649:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066460032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766817.973656:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766817.973659:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766817.973662:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766817.973688:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766817.973703:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766817.973711:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766817/real 1775766817] req@ffff9d81901ae680 x1862023066460160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766817.973721:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066460160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766817.973729:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766817.973731:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766817.973734:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766817.973738:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766817.973751:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766817.973758:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766817/real 1775766817] req@ffff9d81901ad180 x1862023066460288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766817.973767:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066460288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766817.973775:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766817.973777:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766817.973779:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766818.613059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766818.996962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766819.637006:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766820.020979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766820.661067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766821.044940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766821.685021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766822.069042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766822.709033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766823.092955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766823.092968:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766823.092975:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766823.092979:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766823.092986:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3654 00000100:00080000:3.0:1775766823.092995:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766823.093001:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766823.093032:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066460416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766823.093055:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766823.093058:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766823.093059:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766823.093062:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3654 00000100:00080000:3.0:1775766823.093064:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766823.093067:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766823.093074:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766823.093082:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066460544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766823.093089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766823.093091:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766823.093093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766823.093098:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3654 00000100:00080000:3.0:1775766823.093102:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766823.093104:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766823.093108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766823.093113:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066460672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766823.093121:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766823.093123:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766823.093124:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766823.093127:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3654 00000100:00080000:3.0:1775766823.093129:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766823.093137:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066460416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766823.093137:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766823.093148:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766823.093159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066460800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766823.093166:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766823.093169:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766823.093170:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766823.093173:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3654 00000100:00080000:3.0:1775766823.093175:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766823.093177:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766823.093180:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766823.093187:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066460928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775766823.093223:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766823.093275:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766823/real 1775766823] req@ffff9d8188741c00 x1862023066460416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766823.093291:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066460416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766823.093301:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766823.093305:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766823.093309:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066460416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766823.093315:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766823.093331:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766823.093339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766823/real 1775766823] req@ffff9d8188740380 x1862023066460544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766823.093348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066460544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766823.093353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766823.093355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766823.093357:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766823.093379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766823.093390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766823.093396:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766823/real 1775766823] req@ffff9d8188740000 x1862023066460672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766823.093404:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066460672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766823.093409:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766823.093411:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766823.093414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766823.093418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766823.093429:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766823.093444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766823/real 1775766823] req@ffff9d8188742a00 x1862023066460800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766823.093452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066460800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766823.093459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766823.093461:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766823.093463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766823.093467:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766823.093479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766823.093485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766823/real 1775766823] req@ffff9d8188740700 x1862023066460928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766823.093492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066460928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766823.093498:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766823.093499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766823.093501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066460928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766823.733048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766824.116993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766824.757055:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766825.140966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766825.781146:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766826.164982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766826.805060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766827.188996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766827.829115:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766828.212968:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766828.212982:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775766828.212988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766828.212988:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766828.213005:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3659 00000100:00080000:3.0:1775766828.213019:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766828.213029:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766828.213082:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066461056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766828.213112:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766828.213114:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766828.213120:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766828.213124:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3659 00000100:00080000:3.0:1775766828.213127:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766828.213130:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766828.213145:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766828.213156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066461184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766828.213164:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775766828.213169:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066461056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766828.213169:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766828.213170:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766828.213178:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3659 00000100:00080000:3.0:1775766828.213181:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766828.213183:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766828.213186:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766828.213195:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066461312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766828.213202:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766828.213205:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766828.213206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766828.213209:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3659 00000100:00080000:3.0:1775766828.213211:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766828.213213:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766828.213216:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766828.213224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066461440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766828.213229:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766828.213232:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766828.213233:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766828.213236:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3659 00000100:00080000:3.0:1775766828.213238:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766828.213240:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766828.213243:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:1.0:1775766828.213247:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766828.213254:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066461568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766828.213298:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766828/real 1775766828] req@ffff9d8188742300 x1862023066461056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766828.213314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066461056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766828.213322:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766828.213325:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766828.213329:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066461056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766828.213350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766828.213365:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766828.213373:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766828/real 1775766828] req@ffff9d8188741f80 x1862023066461184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766828.213381:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066461184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766828.213387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766828.213389:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766828.213391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766828.213394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766828.213405:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766828.213410:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766828/real 1775766828] req@ffff9d8188743800 x1862023066461312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766828.213417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066461312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766828.213427:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766828.213429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766828.213432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766828.213435:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766828.213445:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766828.213450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766828/real 1775766828] req@ffff9d8188742680 x1862023066461440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766828.213457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066461440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766828.213464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766828.213466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766828.213468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766828.213471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766828.213480:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766828.213485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766828/real 1775766828] req@ffff9d8188741880 x1862023066461568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766828.213492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066461568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766828.213498:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766828.213499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766828.213502:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766828.853093:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766829.236986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766829.877112:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766830.260951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766830.901049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766831.284994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766831.925171:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766832.309045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766832.949071:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766833.332978:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766833.332982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766833.332988:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766833.332993:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766833.333003:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3664 00000100:00080000:3.0:1775766833.333012:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766833.333017:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766833.333051:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066461696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766833.333075:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766833.333078:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766833.333079:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766833.333082:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3664 00000100:00080000:3.0:1775766833.333084:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766833.333087:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766833.333094:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766833.333103:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066461824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766833.333110:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766833.333113:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766833.333114:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766833.333117:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3664 00000100:00080000:3.0:1775766833.333120:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775766833.333122:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066461696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766833.333124:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766833.333132:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766833.333140:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066461952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766833.333151:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766833.333154:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766833.333155:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766833.333158:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3664 00000100:00080000:3.0:1775766833.333173:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766833.333176:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766833.333179:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775766833.333184:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766833.333191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066462080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766833.333198:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766833.333201:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766833.333202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766833.333204:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3664 00000100:00080000:3.0:1775766833.333207:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766833.333209:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766833.333213:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766833.333219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066462208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775766833.333238:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766833/real 1775766833] req@ffff9d8188741180 x1862023066461696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766833.333253:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066461696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766833.333262:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766833.333265:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766833.333268:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066461696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766833.333274:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766833.333289:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766833.333296:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766833/real 1775766833] req@ffff9d8188743100 x1862023066461824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766833.333304:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066461824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766833.333309:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766833.333311:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766833.333313:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766833.333330:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775766833.333341:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766833.333347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766833/real 1775766833] req@ffff9d8188740e00 x1862023066461952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766833.333354:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066461952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766833.333359:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766833.333361:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766833.333363:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066461952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775766833.333367:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766833.333376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766833.333381:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766833/real 1775766833] req@ffff9d8188740a80 x1862023066462080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766833.333388:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066462080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766833.333394:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766833.333396:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766833.333398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775766833.333401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775766833.333411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766833.333416:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766833/real 1775766833] req@ffff9d8188741500 x1862023066462208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775766833.333423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066462208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775766833.333428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775766833.333430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775766833.333432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766833.973039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766834.356960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766834.997025:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766835.380967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766836.021051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766836.404965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766837.045064:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766837.429008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766838.069064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766838.453034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766838.453059:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766838.453070:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766838.453075:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766838.453102:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3669 00000100:00080000:3.0:1775766838.453114:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766838.453121:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766838.453171:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066462336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766838.453205:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766838.453210:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766838.453212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766838.453218:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3669 00000100:00080000:3.0:1775766838.453225:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766838.453232:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766838.453298:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066462336:192.168.202.105@tcp:250:kworker.0 00000800:00000100:0.0:1775766838.453386:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766838.453459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766838/real 1775766838] req@ffff9d8188743480 x1862023066462336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766838.453479:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066462336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766838.453488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766838.453491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766838.453494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066462336:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775766838.454101:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766838.454137:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066462464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766838.454167:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766838.454172:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775766838.454175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766838.454182:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3669 00000100:00080000:2.0:1775766838.454190:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766838.454199:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766838.454206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766838.454224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066462592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766838.454232:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462464:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775766838.454241:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766838.454245:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766838.454247:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766838.454253:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3669 00000100:00080000:2.0:1775766838.454268:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766838.454274:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766838.454279:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775766838.454290:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775766838.454296:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066462720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766838.454305:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766838.454308:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766838.454309:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766838.454317:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3669 00000100:00100000:3.0:1775766838.454318:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766838/real 1775766838] req@ffff9d81984a2680 x1862023066462464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766838.454323:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766838.454333:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766838.454338:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766838.454348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066462464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775766838.454355:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066462848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766838.454362:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.454369:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766838.454375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766838.454406:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766838.454437:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766838.454456:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766838/real 1775766838] req@ffff9d81984a3480 x1862023066462592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.454483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066462592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.454501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.454511:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766838.454517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766838.454530:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766838.454551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766838.454575:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766838/real 1775766838] req@ffff9d81984a2d80 x1862023066462720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.454596:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066462720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.454621:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.454626:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766838.454630:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766838.454639:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766838.454655:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766838.454663:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766838/real 1775766838] req@ffff9d81984a0700 x1862023066462848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766838.454676:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066462848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766838.454682:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766838.454684:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766838.454686:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066462848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766839.093011:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766839.476990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766840.117052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766840.500971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766841.141040:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766841.525010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766842.165082:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766842.548981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766843.189015:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766843.573060:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766843.573121:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766843.573130:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766843.573135:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766843.573146:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3675 00000100:00080000:3.0:1775766843.573170:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766843.573178:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766843.573220:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066462976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766843.573256:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766843.573260:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766843.573262:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766843.573267:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3675 00000100:00080000:3.0:1775766843.573270:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766843.573274:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766843.573284:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766843.573307:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066463104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766843.573319:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766843.573322:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766843.573324:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766843.573328:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3675 00000100:00080000:3.0:1775766843.573331:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766843.573334:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766843.573338:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766843.573350:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066463232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766843.573360:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766843.573364:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766843.573366:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766843.573371:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3675 00000100:00080000:3.0:1775766843.573375:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766843.573383:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066462976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766843.573385:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766843.573399:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766843.573414:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066463360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766843.573426:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766843.573431:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766843.573432:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766843.573436:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3675 00000100:00080000:3.0:1775766843.573461:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766843.573465:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766843.573470:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766843.573473:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766843.573491:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066463488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766843.573538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766843/real 1775766843] req@ffff9d8188742d80 x1862023066462976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766843.573625:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066462976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766843.573638:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766843.573642:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766843.573647:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066462976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766843.573656:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766843.573684:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766843.573698:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766843/real 1775766843] req@ffff9d8188743b80 x1862023066463104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766843.573711:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066463104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766843.573720:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766843.573724:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766843.573727:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766843.573734:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766843.573750:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766843.573760:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766843/real 1775766843] req@ffff9d8188743480 x1862023066463232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766843.573772:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066463232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766843.573780:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766843.573783:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766843.573786:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766843.573819:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766843.573855:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766843.573867:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766843/real 1775766843] req@ffff9d8188741500 x1862023066463360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766843.573880:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066463360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766843.573930:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766843.573935:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766843.573939:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766843.573946:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766843.573970:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766843.573983:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766843/real 1775766843] req@ffff9d8188740a80 x1862023066463488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766843.573997:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066463488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766843.574007:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766843.574011:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766843.574014:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766844.213125:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766844.597030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766845.237125:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766845.621020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766846.261063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766846.644989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766847.285079:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766847.668952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766848.309145:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766848.693017:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766848.693023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766848.693039:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766848.693048:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766848.693069:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3680 00000100:00080000:3.0:1775766848.693089:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766848.693100:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766848.693163:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066463616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766848.693209:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766848.693214:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766848.693217:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766848.693222:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3680 00000100:00080000:3.0:1775766848.693227:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766848.693231:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766848.693245:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766848.693262:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066463744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766848.693287:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766848.693296:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066463616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766848.693296:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766848.693300:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766848.693315:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3680 00000100:00080000:3.0:1775766848.693319:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766848.693322:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766848.693329:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766848.693346:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066463872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766848.693358:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766848.693363:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766848.693365:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766848.693370:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3680 00000100:00080000:3.0:1775766848.693393:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766848.693398:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766848.693404:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766848.693412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766848.693421:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066464000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766848.693435:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766848.693441:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766848.693444:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766848.693448:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3680 00000100:00080000:3.0:1775766848.693454:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766848.693459:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766848.693465:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766848.693480:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066464128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766848.693532:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766848/real 1775766848] req@ffff9d8188740e00 x1862023066463616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766848.693574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066463616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766848.693597:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766848.693604:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766848.693609:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066463616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766848.693644:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766848.693689:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766848.693714:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766848/real 1775766848] req@ffff9d8188743100 x1862023066463744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766848.693755:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066463744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766848.693772:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766848.693778:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766848.693782:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766848.693791:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766848.693821:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766848.693835:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766848/real 1775766848] req@ffff9d8188741180 x1862023066463872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766848.693850:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066463872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766848.693861:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766848.693865:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766848.693869:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066463872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766848.693917:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766848.693944:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766848.693956:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766848/real 1775766848] req@ffff9d8188741880 x1862023066464000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766848.693971:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066464000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766848.693991:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766848.693996:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766848.694000:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766848.694007:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766848.694026:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766848.694036:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766848/real 1775766848] req@ffff9d8188742680 x1862023066464128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766848.694050:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066464128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766848.694061:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766848.694066:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766848.694072:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766849.333144:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766849.716968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766850.357071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766850.741065:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766851.381071:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766851.765021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766852.405130:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766852.789014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766853.429063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766853.812955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766853.812966:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766853.812974:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766853.812979:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766853.812989:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3685 00000100:00080000:3.0:1775766853.812999:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766853.813006:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766853.813044:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066464256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766853.813072:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766853.813076:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766853.813078:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766853.813082:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3685 00000100:00080000:3.0:1775766853.813087:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766853.813092:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766853.813102:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766853.813111:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066464256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766853.813118:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066464384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766853.813137:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766853.813140:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766853.813143:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766853.813147:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3685 00000100:00080000:3.0:1775766853.813161:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766853.813167:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766853.813168:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766853.813176:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766853.813190:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066464512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766853.813200:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766853.813204:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766853.813206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766853.813210:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766853/real 1775766853] req@ffff9d8188743800 x1862023066464256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766853.813214:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3685 00000100:00080000:3.0:1775766853.813221:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766853.813228:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766853.813233:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066464256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775766853.813234:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775766853.813251:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775766853.813252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066464640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766853.813257:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766853.813264:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066464256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766853.813275:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766853.813278:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766853.813280:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766853.813285:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3685 00000100:00100000:0.0:1775766853.813289:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464384:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775766853.813290:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766853.813297:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766853.813303:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766853.813310:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766853.813318:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066464768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766853.813325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766853/real 1775766853] req@ffff9d8188741f80 x1862023066464384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766853.813344:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066464384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766853.813364:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766853.813369:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766853.813372:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766853.813379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766853.813398:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766853.813406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766853/real 1775766853] req@ffff9d8188742300 x1862023066464512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766853.813414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066464512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766853.813428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766853.813430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766853.813433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766853.813437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766853.813470:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766853.813481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766853/real 1775766853] req@ffff9d8188740700 x1862023066464640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766853.813495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066464640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766853.813505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766853.813509:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766853.813511:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766853.813523:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766853.813538:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766853.813545:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766853/real 1775766853] req@ffff9d8188742a00 x1862023066464768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766853.813553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066464768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766853.813559:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766853.813561:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766853.813563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066464768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766854.453024:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766854.836986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766855.477078:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766855.860971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766856.501033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766856.884998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766857.525015:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766857.908949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766858.549012:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766858.932970:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766858.932973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766858.932981:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766858.932985:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766858.932993:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3690 00000100:00080000:3.0:1775766858.933002:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766858.933008:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766858.933041:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066464896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766858.933082:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766858.933085:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766858.933086:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766858.933089:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3690 00000100:00080000:3.0:1775766858.933092:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766858.933095:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766858.933103:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766858.933112:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066465024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766858.933121:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066464896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766858.933123:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766858.933130:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766858.933131:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766858.933135:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3690 00000100:00080000:3.0:1775766858.933138:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766858.933140:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766858.933143:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766858.933152:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066465152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766858.933158:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766858.933161:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766858.933162:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766858.933165:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3690 00000100:00080000:3.0:1775766858.933174:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766858.933184:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766858.933187:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766858.933196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066465280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766858.933202:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766858.933205:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766858.933206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766858.933208:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3690 00000100:00080000:3.0:1775766858.933210:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766858.933213:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766858.933217:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766858.933226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066465408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766858.933239:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766858.933261:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766858.933275:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766858.933292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766859.573155:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766859.956987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766860.597044:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766860.980997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766861.621046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766862.004973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775766862.005170:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775766862.005203:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766862.005219:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766862.005259:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766862.005266:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766862.005271:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766862.005277:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766862.005329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766858/real 1775766862] req@ffff9d8188740380 x1862023066465024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766862.005405:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066465024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766862.005416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766862.005420:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766862.005423:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766862.005430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766858/real 1775766862] req@ffff9d8188741c00 x1862023066465152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766862.005448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066465152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766862.005473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766862.005479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766862.005482:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766862.005489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766858/real 1775766862] req@ffff9d8188742a00 x1862023066465280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766862.005503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066465280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766862.005513:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766862.005516:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766862.005519:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766862.005542:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766858/real 1775766862] req@ffff9d8188740700 x1862023066465408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766862.005553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066465408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766862.005563:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766862.005567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766862.005570:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465408:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766862.005576:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766858/real 1775766862] req@ffff9d8188740000 x1862023066464896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766862.005588:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066464896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766862.005595:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766862.005599:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766862.005601:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066464896:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775766862.645099:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766863.029035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766863.669051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766864.052970:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766864.052971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766864.052983:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766864.052988:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766864.053002:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3695 00000100:00080000:3.0:1775766864.053022:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766864.053032:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766864.053083:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066465536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766864.053122:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766864.053125:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766864.053126:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766864.053131:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3695 00000100:00080000:3.0:1775766864.053134:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766864.053136:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766864.053147:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766864.053156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066465664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766864.053164:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766864.053168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775766864.053171:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066465536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766864.053171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766864.053178:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3695 00000100:00080000:3.0:1775766864.053181:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766864.053184:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766864.053187:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766864.053196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066465792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766864.053202:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766864.053205:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766864.053206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766864.053209:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3695 00000100:00080000:3.0:1775766864.053212:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766864.053215:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766864.053220:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766864.053233:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066465920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766864.053242:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766864.053246:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766864.053248:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766864.053253:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3695 00000100:00080000:3.0:1775766864.053257:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766864.053263:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766864.053266:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766864.053272:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766864.053294:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066466048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766864.053356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766864/real 1775766864] req@ffff9d8188742300 x1862023066465536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766864.053380:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066465536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766864.053393:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766864.053398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766864.053403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066465536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766864.053434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766864.053463:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766864.053480:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766864/real 1775766864] req@ffff9d8188741f80 x1862023066465664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766864.053493:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066465664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766864.053502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766864.053505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766864.053509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766864.053516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766864.053539:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766864.053550:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766864/real 1775766864] req@ffff9d8188743800 x1862023066465792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766864.053563:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066465792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766864.053572:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766864.053575:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766864.053578:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766864.053585:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766864.053607:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766864.053619:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766864/real 1775766864] req@ffff9d8188742680 x1862023066465920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766864.053631:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066465920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766864.053641:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766864.053644:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766864.053647:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066465920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766864.053652:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766864.053670:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766864.053681:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766864/real 1775766864] req@ffff9d8188741880 x1862023066466048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766864.053702:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066466048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766864.053719:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766864.053723:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766864.053726:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766864.693033:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766865.077028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766865.717007:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766866.100972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766866.741063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766867.124972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766867.765032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766868.148946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766868.789069:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766869.172976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766869.172989:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766869.172995:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766869.172998:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766869.173007:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3700 00000100:00080000:3.0:1775766869.173016:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766869.173022:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766869.173054:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066466176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766869.173077:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766869.173079:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766869.173081:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766869.173083:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3700 00000100:00080000:3.0:1775766869.173085:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766869.173088:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766869.173096:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766869.173104:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066466304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766869.173112:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766869.173117:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066466176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766869.173117:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766869.173120:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766869.173128:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3700 00000100:00080000:3.0:1775766869.173130:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766869.173132:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766869.173135:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766869.173144:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066466432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766869.173150:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766869.173153:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766869.173154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766869.173158:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3700 00000100:00080000:3.0:1775766869.173168:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766869.173170:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766869.173174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766869.173179:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766869.173184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066466560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766869.173191:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766869.173194:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766869.173195:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766869.173198:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3700 00000100:00080000:3.0:1775766869.173201:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766869.173203:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766869.173205:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766869.173212:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066466688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766869.173238:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766869/real 1775766869] req@ffff9d8188741180 x1862023066466176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766869.173258:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066466176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766869.173265:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766869.173268:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766869.173271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066466176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766869.173315:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766869.173337:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766869.173351:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766869/real 1775766869] req@ffff9d8188743100 x1862023066466304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766869.173372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066466304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766869.173382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766869.173385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766869.173388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766869.173393:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766869.173421:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766869.173433:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766869/real 1775766869] req@ffff9d8188740e00 x1862023066466432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766869.173446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066466432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766869.173456:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766869.173459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766869.173463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766869.173469:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766869.173487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766869.173494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766869/real 1775766869] req@ffff9d8188740a80 x1862023066466560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766869.173502:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066466560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766869.173513:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766869.173515:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766869.173517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766869.173520:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766869.173530:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766869.173535:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766869/real 1775766869] req@ffff9d8188741500 x1862023066466688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766869.173542:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066466688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766869.173548:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766869.173549:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766869.173551:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766869.813099:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766870.196954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766870.837080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766871.220977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766871.861038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766872.244972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766872.885042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766873.268945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766873.909117:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766874.292995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766874.293016:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766874.293021:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766874.293025:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766874.293033:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3705 00000100:00080000:0.0:1775766874.293042:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766874.293048:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766874.293083:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066466816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766874.293114:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766874.293118:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766874.293120:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766874.293124:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3705 00000100:00080000:0.0:1775766874.293129:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766874.293134:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766874.293145:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766874.293153:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066466816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766874.293161:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066466944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766874.293174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766874.293178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766874.293180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766874.293185:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3705 00000100:00080000:0.0:1775766874.293189:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766874.293193:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766874.293197:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766874.293207:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066467072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766874.293214:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766874.293216:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766874.293217:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766874.293221:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3705 00000100:00080000:0.0:1775766874.293225:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775766874.293226:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766874.293246:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766874.293253:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766874.293268:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066467200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766874.293279:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766874.293283:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766874/real 1775766874] req@ffff9d81901ad180 x1862023066466816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766874.293285:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766874.293290:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766874.293299:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3705 00000100:00080000:0.0:1775766874.293303:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766874.293307:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066466816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766874.293308:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766874.293316:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775766874.293321:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766874.293326:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766874.293329:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066467328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766874.293333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066466816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766874.293365:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766874.293383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766874.293391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766874/real 1775766874] req@ffff9d81901ae680 x1862023066466944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766874.293400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066466944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766874.293405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766874.293408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766874.293410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066466944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766874.293414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766874.293426:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766874.293432:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766874/real 1775766874] req@ffff9d81901aed80 x1862023066467072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766874.293438:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066467072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766874.293444:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766874.293446:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766874.293448:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766874.293451:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766874.293461:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766874.293466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766874/real 1775766874] req@ffff9d81901afb80 x1862023066467200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766874.293473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066467200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766874.293479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766874.293481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766874.293482:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766874.293486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766874.293496:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766874.293501:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766874/real 1775766874] req@ffff9d81901af100 x1862023066467328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766874.293513:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066467328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766874.293518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766874.293520:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766874.293521:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766874.933051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766875.316984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766875.957067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766876.340991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766876.981069:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766877.364979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766878.005058:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766878.389007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766879.029097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766879.412999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766879.413013:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766879.413019:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766879.413023:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766879.413030:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3710 00000100:00080000:3.0:1775766879.413039:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766879.413044:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766879.413076:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066467456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766879.413101:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766879.413103:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766879.413104:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766879.413107:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3710 00000100:00080000:3.0:1775766879.413109:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766879.413112:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766879.413119:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766879.413129:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066467584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766879.413136:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766879.413138:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766879.413140:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766879.413143:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3710 00000100:00080000:3.0:1775766879.413144:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766879.413149:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766879.413152:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066467456:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775766879.413152:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766879.413161:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066467712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766879.413168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766879.413170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766879.413171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766879.413174:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3710 00000100:00080000:3.0:1775766879.413175:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766879.413190:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766879.413193:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766879.413203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066467840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766879.413210:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766879.413212:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766879.413213:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766879.413216:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3710 00000100:00080000:3.0:1775766879.413221:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766879.413223:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766879.413226:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775766879.413233:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766879.413235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066467968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766879.413281:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766879/real 1775766879] req@ffff9d8188743480 x1862023066467456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766879.413302:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066467456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766879.413314:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766879.413318:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766879.413322:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066467456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766879.413329:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766879.413351:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766879.413364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766879/real 1775766879] req@ffff9d8188743b80 x1862023066467584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766879.413378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066467584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766879.413387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766879.413391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766879.413394:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766879.413417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766879.413436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766879.413444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766879/real 1775766879] req@ffff9d8188742d80 x1862023066467712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766879.413452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066467712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766879.413458:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766879.413460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766879.413463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766879.413467:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766879.413477:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766879.413483:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766879/real 1775766879] req@ffff9d8188741500 x1862023066467840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766879.413490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066467840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766879.413496:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766879.413497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766879.413499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766879.413502:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766879.413512:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766879.413517:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766879/real 1775766879] req@ffff9d8188740a80 x1862023066467968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766879.413524:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066467968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766879.413530:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766879.413531:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766879.413533:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066467968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766880.053024:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766880.436993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766881.077045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766881.460972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766882.101044:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766882.484983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766883.125062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766883.508990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766884.149077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766884.533013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766884.533020:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766884.533032:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766884.533038:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766884.533051:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3716 00000100:00080000:3.0:1775766884.533066:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766884.533074:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766884.533151:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066468096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766884.533179:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766884.533184:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766884.533186:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766884.533190:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3716 00000100:00080000:3.0:1775766884.533194:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766884.533199:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766884.533211:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766884.533218:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066468096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766884.533224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066468224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766884.533246:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766884.533250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766884.533252:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766884.533256:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3716 00000100:00080000:3.0:1775766884.533259:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766884.533263:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766884.533268:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766884.533278:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066468352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766884.533287:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766884.533290:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766884.533292:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766884.533295:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766884.533296:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3716 00000100:00080000:3.0:1775766884.533299:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766884.533302:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766884.533307:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766884.533315:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066468480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766884.533324:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766884.533327:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766884.533329:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766884.533332:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3716 00000100:00080000:3.0:1775766884.533336:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766884.533340:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766884.533344:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766884.533352:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066468608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766884.533355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766884/real 1775766884] req@ffff9d8188740e00 x1862023066468096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766884.533375:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066468096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766884.533387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766884.533391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766884.533396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066468096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766884.533422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766884.533441:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766884.533451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766884/real 1775766884] req@ffff9d8188743100 x1862023066468224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766884.533459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066468224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766884.533465:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766884.533468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766884.533470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766884.533474:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766884.533484:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766884.533490:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766884/real 1775766884] req@ffff9d8188741180 x1862023066468352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766884.533498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066468352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766884.533503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766884.533505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766884.533506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766884.533510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766884.533522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766884.533527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766884/real 1775766884] req@ffff9d8188741880 x1862023066468480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766884.533535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066468480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766884.533541:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766884.533543:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766884.533545:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766884.533548:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766884.533557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766884.533562:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766884/real 1775766884] req@ffff9d8188742680 x1862023066468608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766884.533569:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066468608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766884.533580:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766884.533582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766884.533584:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766885.173074:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766885.557005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766886.197099:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766886.580997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766887.221060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766887.605042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766888.245138:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766888.629170:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766889.269105:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766889.652987:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766889.652990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766889.652998:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766889.653002:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766889.653011:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3721 00000100:00080000:3.0:1775766889.653020:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766889.653026:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766889.653060:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066468736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766889.653100:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766889.653103:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766889.653104:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766889.653107:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3721 00000100:00080000:3.0:1775766889.653110:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766889.653112:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766889.653119:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766889.653129:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066468864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766889.653136:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766889.653139:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766889.653140:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766889.653144:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3721 00000100:00080000:3.0:1775766889.653146:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766889.653148:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766889.653151:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766889.653156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066468992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766889.653163:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766889.653167:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775766889.653168:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066468736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766889.653169:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766889.653174:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3721 00000100:00080000:3.0:1775766889.653178:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766889.653181:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766889.653184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766889.653193:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066469120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766889.653199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766889.653202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766889.653203:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766889.653206:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3721 00000100:00080000:3.0:1775766889.653208:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766889.653211:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766889.653214:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766889.653220:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066469248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766889.653239:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766889.653284:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766889/real 1775766889] req@ffff9d8188743800 x1862023066468736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766889.653304:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066468736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766889.653315:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766889.653320:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766889.653324:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066468736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766889.653332:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766889.653356:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766889.653370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766889/real 1775766889] req@ffff9d8188741f80 x1862023066468864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766889.653383:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066468864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766889.653397:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766889.653401:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766889.653405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766889.653428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766889.653449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766889.653461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766889/real 1775766889] req@ffff9d8188742300 x1862023066468992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766889.653473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066468992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766889.653482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766889.653486:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766889.653489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066468992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766889.653495:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766889.653515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766889.653527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766889/real 1775766889] req@ffff9d8188740000 x1862023066469120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766889.653539:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066469120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766889.653549:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766889.653552:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766889.653556:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766889.653562:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766889.653581:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766889.653591:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766889/real 1775766889] req@ffff9d8188740700 x1862023066469248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766889.653604:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066469248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766889.653612:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766889.653615:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766889.653618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766890.293109:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766890.676993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766891.317091:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766891.700948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766892.341167:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766892.724959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766893.365054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766893.749114:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766894.389089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766894.773005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766894.773064:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766894.773071:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766894.773075:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766894.773084:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3726 00000100:00080000:1.0:1775766894.773095:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766894.773101:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766894.773140:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066469376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766894.773168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766894.773171:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766894.773173:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766894.773182:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3726 00000100:00080000:1.0:1775766894.773185:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766894.773189:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766894.773199:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766894.773212:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066469504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766894.773221:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766894.773225:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766894.773226:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766894.773231:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3726 00000100:00080000:1.0:1775766894.773234:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766894.773238:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066469376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766894.773239:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766894.773248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766894.773257:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066469632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766894.773266:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766894.773270:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766894.773271:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766894.773274:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3726 00000100:00080000:1.0:1775766894.773277:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766894.773279:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766894.773284:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766894.773294:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066469760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766894.773301:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766894.773305:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766894.773306:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766894.773309:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3726 00000100:00080000:1.0:1775766894.773314:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766894.773320:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766894.773323:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766894.773328:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766894.773339:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066469888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766894.773381:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766894/real 1775766894] req@ffff9d8190553480 x1862023066469376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766894.773424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066469376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766894.773441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766894.773447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766894.773453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066469376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766894.773469:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766894.773499:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766894.773515:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766894/real 1775766894] req@ffff9d8190551880 x1862023066469504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766894.773534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066469504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766894.773563:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766894.773567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766894.773571:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766894.773598:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766894.773636:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766894.773648:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766894/real 1775766894] req@ffff9d8190551500 x1862023066469632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766894.773665:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066469632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766894.773677:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766894.773699:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766894.773716:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766894.773723:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766894.773744:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766894.773772:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766894/real 1775766894] req@ffff9d8190553100 x1862023066469760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766894.773788:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066469760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766894.773802:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766894.773806:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766894.773811:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766894.773818:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766894.773836:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766894.773847:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766894/real 1775766894] req@ffff9d8190550e00 x1862023066469888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766894.773863:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066469888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766894.773875:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766894.773879:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766894.773883:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066469888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766895.413049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766895.797074:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766896.437143:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766896.820989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766897.461026:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766897.845019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766898.485155:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766898.868929:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766899.509206:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775766899.893034:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766899.893036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766899.893053:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766899.893057:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766899.893076:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3731 00000100:00080000:1.0:1775766899.893090:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766899.893097:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766899.893134:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066470016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766899.893174:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766899.893178:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766899.893179:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766899.893182:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3731 00000100:00080000:1.0:1775766899.893184:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766899.893187:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766899.893194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766899.893202:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066470144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766899.893211:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766899.893213:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766899.893214:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766899.893217:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3731 00000100:00080000:1.0:1775766899.893219:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766899.893222:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766899.893225:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766899.893232:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066470272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766899.893239:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766899.893242:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766899.893245:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766899.893248:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3731 00000100:00080000:1.0:1775766899.893250:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766899.893253:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766899.893256:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766899.893263:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066470400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766899.893278:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766899.893283:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766899.893284:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766899.893299:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3731 00000100:00100000:0.0:1775766899.893302:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066470016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766899.893303:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766899.893319:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766899.893327:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766899.893336:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066470528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775766899.893483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766899.893590:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766899/real 1775766899] req@ffff9d8190550e00 x1862023066470016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766899.893646:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066470016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766899.893669:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766899.893678:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766899.893685:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066470016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766899.893731:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766899.893776:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766899.893799:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766899/real 1775766899] req@ffff9d8190553100 x1862023066470144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766899.893814:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066470144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766899.893842:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766899.893847:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766899.893851:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766899.893946:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766899.894004:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766899.894021:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766899/real 1775766899] req@ffff9d8190551500 x1862023066470272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766899.894036:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066470272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766899.894055:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766899.894059:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766899.894063:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766899.894099:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766899.894123:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766899.894135:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766899/real 1775766899] req@ffff9d8190551880 x1862023066470400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766899.894175:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066470400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766899.894191:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766899.894197:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766899.894203:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766899.894214:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766899.894249:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766899.894269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766899/real 1775766899] req@ffff9d8190553480 x1862023066470528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766899.894304:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066470528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766899.894316:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766899.894320:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766899.894325:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766900.533177:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766900.916992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766901.557130:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766901.940989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766902.581112:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766902.964969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766903.605217:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766903.989093:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766904.629017:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766905.012996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766905.013029:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766905.013035:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766905.013038:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766905.013045:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3736 00000100:00080000:3.0:1775766905.013054:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766905.013059:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766905.013092:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066470656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766905.013117:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766905.013120:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766905.013121:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766905.013124:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3736 00000100:00080000:3.0:1775766905.013127:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766905.013131:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766905.013140:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766905.013153:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066470784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766905.013167:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066470656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766905.013174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766905.013178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766905.013180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766905.013185:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3736 00000100:00080000:3.0:1775766905.013189:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766905.013193:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766905.013199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766905.013210:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066470912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766905.013220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766905.013226:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775766905.013227:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766905.013229:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766905.013235:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3736 00000100:00080000:3.0:1775766905.013239:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766905.013243:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766905.013249:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766905.013262:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066471040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766905.013272:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766905.013274:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766905.013276:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766905.013281:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766905/real 1775766905] req@ffff9d8188742a00 x1862023066470656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766905.013281:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3736 00000100:00080000:3.0:1775766905.013287:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766905.013293:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766905.013300:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766905.013310:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066471168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766905.013314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066470656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766905.013327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766905.013331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766905.013335:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066470656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766905.013353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766905.013380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766905.013390:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766905/real 1775766905] req@ffff9d8188741c00 x1862023066470784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766905.013398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066470784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766905.013404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766905.013407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766905.013409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766905.013416:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766905.013437:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766905.013448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766905/real 1775766905] req@ffff9d8188740380 x1862023066470912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766905.013462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066470912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766905.013470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766905.013474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766905.013477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066470912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766905.013484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766905.013505:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766905.013527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766905/real 1775766905] req@ffff9d8188740700 x1862023066471040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766905.013540:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066471040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766905.013550:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766905.013553:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766905.013556:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766905.013562:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766905.013575:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766905.013581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766905/real 1775766905] req@ffff9d8188740000 x1862023066471168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766905.013589:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066471168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766905.013603:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766905.013605:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766905.013607:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766905.653049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766906.036999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766906.677084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766907.061023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766907.701195:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766908.085020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766908.725113:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766909.108955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766909.749179:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766910.132999:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766910.133002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766910.133014:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766910.133021:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766910.133037:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3741 00000100:00080000:3.0:1775766910.133050:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766910.133059:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766910.133110:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066471296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766910.133141:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766910.133145:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766910.133146:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766910.133149:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3741 00000100:00080000:3.0:1775766910.133153:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766910.133156:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766910.133167:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766910.133177:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066471424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766910.133196:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766910.133197:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066471296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766910.133204:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766910.133208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766910.133215:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3741 00000100:00080000:3.0:1775766910.133219:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766910.133223:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766910.133229:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766910.133253:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066471552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766910.133264:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766910.133268:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766910.133269:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766910.133272:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3741 00000100:00080000:3.0:1775766910.133283:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766910.133288:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766910.133294:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766910.133299:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766910.133314:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066471680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766910.133326:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766910.133330:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766910.133333:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766910.133339:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3741 00000100:00080000:3.0:1775766910.133344:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766910.133348:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766910.133353:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766910.133363:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066471808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766910.133391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766910/real 1775766910] req@ffff9d8188742300 x1862023066471296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766910.133411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066471296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766910.133422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766910.133427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766910.133432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066471296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766910.133463:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766910.133488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766910.133499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766910/real 1775766910] req@ffff9d8188741f80 x1862023066471424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766910.133514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066471424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766910.133521:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766910.133523:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766910.133526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766910.133533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766910.133553:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766910.133565:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766910/real 1775766910] req@ffff9d8188743800 x1862023066471552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766910.133579:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066471552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766910.133588:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766910.133592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766910.133594:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766910.133599:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766910.133616:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766910.133624:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766910/real 1775766910] req@ffff9d8188742680 x1862023066471680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766910.133631:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066471680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766910.133645:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766910.133649:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766910.133652:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766910.133658:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766910.133679:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766910.133690:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766910/real 1775766910] req@ffff9d8188741880 x1862023066471808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766910.133703:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066471808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766910.133710:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766910.133713:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766910.133715:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066471808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766910.773176:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766911.157299:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766911.797042:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766912.181042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766912.821105:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766913.204991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766913.845191:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766914.228969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766914.869026:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766915.252969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766915.253001:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766915.253007:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766915.253011:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766915.253019:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3746 00000100:00080000:0.0:1775766915.253027:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766915.253033:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766915.253067:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066471936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766915.253093:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766915.253096:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766915.253097:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766915.253099:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3746 00000100:00080000:0.0:1775766915.253102:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766915.253105:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766915.253113:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766915.253123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066472064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766915.253127:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066471936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766915.253141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766915.253144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766915.253146:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766915.253151:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3746 00000100:00080000:0.0:1775766915.253155:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766915.253160:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766915.253166:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766915.253178:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066472192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766915.253189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766915.253193:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766915.253195:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766915.253200:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766915.253203:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3746 00000100:00080000:0.0:1775766915.253208:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766915.253211:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766915.253216:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766915.253226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066472320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766915.253233:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766915.253235:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766915.253237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766915.253240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3746 00000100:00080000:0.0:1775766915.253243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766915.253246:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766915.253249:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766915.253260:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066472448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766915.253260:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766915/real 1775766915] req@ffff9d81901af100 x1862023066471936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766915.253294:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066471936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766915.253303:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766915.253306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766915.253309:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066471936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766915.253327:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766915.253344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766915.253352:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766915/real 1775766915] req@ffff9d81901afb80 x1862023066472064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766915.253360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066472064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766915.253370:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766915.253372:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766915.253375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766915.253378:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766915.253388:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766915.253393:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766915/real 1775766915] req@ffff9d81901aed80 x1862023066472192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766915.253401:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066472192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766915.253406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766915.253407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766915.253409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766915.253413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766915.253422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766915.253428:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766915/real 1775766915] req@ffff9d81901ae680 x1862023066472320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766915.253435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066472320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766915.253440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766915.253442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766915.253445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766915.253448:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766915.253457:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766915.253462:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766915/real 1775766915] req@ffff9d81901ad180 x1862023066472448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766915.253469:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066472448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766915.253475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766915.253476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766915.253478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766915.893072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766916.276961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766916.917021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766917.300939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766917.941137:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766918.324996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766918.965118:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766919.348971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766919.989125:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766920.373006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766920.373072:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766920.373085:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766920.373092:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766920.373108:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3751 00000100:00080000:3.0:1775766920.373124:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766920.373136:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766920.373197:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066472576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766920.373243:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766920.373248:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766920.373251:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766920.373257:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3751 00000100:00080000:3.0:1775766920.373264:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766920.373271:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766920.373286:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766920.373298:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066472576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766920.373309:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066472704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766920.373326:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766920.373331:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766920.373333:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766920.373339:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3751 00000100:00080000:3.0:1775766920.373343:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766920.373348:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766920.373354:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766920.373368:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066472832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766920.373380:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775766920.373390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766920.373391:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766920.373397:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766920.373403:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3751 00000100:00080000:3.0:1775766920.373409:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766920.373414:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766920.373421:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766920.373450:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066472960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766920.373460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766920/real 1775766920] req@ffff9d8188741180 x1862023066472576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766920.373525:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766920.373530:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766920.373532:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766920.373538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066472576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766920.373543:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3751 00000100:00080000:3.0:1775766920.373554:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766920.373565:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766920.373576:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775766920.373596:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775766920.373599:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066473088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766920.373609:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766920.373626:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066472576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766920.373662:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766920.373692:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766920.373706:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766920/real 1775766920] req@ffff9d8188743100 x1862023066472704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766920.373722:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066472704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766920.373732:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766920.373737:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766920.373741:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766920.373747:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766920.373812:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766920.373833:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766920/real 1775766920] req@ffff9d8188740e00 x1862023066472832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766920.373857:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066472832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766920.373913:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766920.373920:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766920.373926:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766920.373935:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766920.373965:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766920.373980:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766920/real 1775766920] req@ffff9d8188740a80 x1862023066472960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766920.373996:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066472960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766920.374007:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766920.374010:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766920.374024:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066472960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766920.374031:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766920.374071:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766920.374082:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766920/real 1775766920] req@ffff9d8188741500 x1862023066473088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766920.374096:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066473088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766920.374105:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766920.374108:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766920.374111:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766921.013092:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766921.396935:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766922.037093:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766922.421032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766923.061076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766923.444976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766924.085076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766924.469033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766925.109109:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766925.492951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766925.492952:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766925.492960:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766925.492964:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766925.492972:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3757 00000100:00080000:3.0:1775766925.492980:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766925.492986:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766925.493025:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066473216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766925.493047:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766925.493050:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766925.493051:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766925.493054:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3757 00000100:00080000:3.0:1775766925.493057:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766925.493060:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766925.493067:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766925.493074:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066473344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766925.493080:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766925.493083:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766925.493084:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766925.493088:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3757 00000100:00080000:3.0:1775766925.493090:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766925.493092:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766925.493095:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766925.493100:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066473472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766925.493107:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766925.493109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766925.493110:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766925.493123:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3757 00000100:00080000:3.0:1775766925.493125:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766925.493129:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766925.493133:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766925.493143:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066473600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766925.493161:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766925.493167:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766925.493170:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775766925.493172:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066473216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766925.493179:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3757 00000100:00080000:3.0:1775766925.493187:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766925.493191:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766925.493196:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766925.493221:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066473728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766925.493351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766925.493379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766925.493407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766925.493424:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766926.133123:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766926.517014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766927.157080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766927.540992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766928.181087:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766928.564994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775766928.565239:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775766928.565298:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775766928.565320:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766928.565378:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766928.565390:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766928.565401:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775766928.565411:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766928.565518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766925/real 1775766928] req@ffff9d8188741500 x1862023066473600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766928.565553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066473600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766928.565570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766928.565575:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766928.565580:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766928.565598:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766925/real 1775766928] req@ffff9d8188740a80 x1862023066473728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766928.565612:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066473728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766928.565621:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766928.565625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766928.565629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473728:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766928.565636:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766925/real 1775766928] req@ffff9d8188742d80 x1862023066473216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766928.565655:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066473216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766928.565697:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766928.565702:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766928.565706:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066473216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766928.565715:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766925/real 1775766928] req@ffff9d8188743b80 x1862023066473344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766928.565730:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066473344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766928.565738:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766928.565742:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766928.565746:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766928.565753:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766925/real 1775766928] req@ffff9d8188743480 x1862023066473472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766928.565764:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066473472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766928.565772:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766928.565776:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766928.565779:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473472:192.168.202.105@tcp:38:kworker.0 00010000:00010000:0.0:1775766929.205082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766929.588989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766930.229128:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766930.613058:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766930.613058:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766930.613073:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766930.613079:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766930.613099:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3762 00000100:00080000:3.0:1775766930.613114:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766930.613124:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766930.613175:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066473856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766930.613227:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766930.613231:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766930.613233:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766930.613237:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3762 00000100:00080000:3.0:1775766930.613242:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766930.613248:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766930.613263:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766930.613287:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066473856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766930.613287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066473984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766930.613311:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766930.613316:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766930.613319:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766930.613326:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3762 00000100:00080000:3.0:1775766930.613331:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766930.613337:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766930.613345:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766930.613371:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066474112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766930.613384:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775766930.613393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766930.613395:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766930.613399:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766930.613407:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3762 00000100:00080000:3.0:1775766930.613414:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766930.613421:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766930.613428:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766930.613446:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066474240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766930.613467:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766930.613474:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766930/real 1775766930] req@ffff9d8188740e00 x1862023066473856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766930.613474:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766930.613479:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766930.613492:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3762 00000100:00080000:3.0:1775766930.613502:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766930.613509:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766930.613517:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766930.613523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066473856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766930.613537:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066474368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766930.613540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766930.613555:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766930.613566:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066473856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766930.613606:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766930.613661:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766930.613680:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766930/real 1775766930] req@ffff9d8188743100 x1862023066473984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766930.613696:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066473984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766930.613706:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766930.613711:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766930.613715:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066473984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766930.613723:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766930.613754:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766930.613769:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766930/real 1775766930] req@ffff9d8188741180 x1862023066474112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766930.613786:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066474112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766930.613797:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766930.613802:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766930.613807:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766930.613816:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766930.613843:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766930.613859:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766930/real 1775766930] req@ffff9d8188741880 x1862023066474240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766930.613907:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066474240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766930.613921:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766930.613926:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766930.613930:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766930.613937:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766930.613965:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766930.613980:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766930/real 1775766930] req@ffff9d8188742680 x1862023066474368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766930.614003:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066474368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766930.614014:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766930.614017:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766930.614021:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766931.253125:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766931.637008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766932.277074:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766932.660987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766933.301028:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766933.684976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766934.325068:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766934.708955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766935.349093:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766935.732974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766935.732994:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766935.733004:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766935.733010:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766935.733019:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3767 00000100:00080000:0.0:1775766935.733031:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766935.733039:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766935.733126:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066474496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766935.733153:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766935.733156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766935.733158:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766935.733161:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3767 00000100:00080000:0.0:1775766935.733172:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766935.733177:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766935.733188:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766935.733197:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066474496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766935.733208:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066474624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766935.733218:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766935.733221:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766935.733223:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766935.733228:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3767 00000100:00080000:0.0:1775766935.733231:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766935.733233:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766935.733239:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766935.733253:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066474752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766935.733263:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766935.733267:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766935.733269:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775766935.733274:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775766935.733276:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3767 00000100:00080000:0.0:1775766935.733285:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766935.733290:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766935.733298:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766935.733315:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066474880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766935.733327:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766935.733328:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766935/real 1775766935] req@ffff9d81901ad180 x1862023066474496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766935.733337:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766935.733342:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766935.733355:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3767 00000100:00080000:0.0:1775766935.733360:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766935.733366:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766935.733367:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066474496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775766935.733379:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775766935.733384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766935.733388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766935.733392:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066475008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766935.733395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066474496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766935.733438:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766935.733456:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766935.733465:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766935/real 1775766935] req@ffff9d81901ae680 x1862023066474624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766935.733473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066474624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766935.733479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766935.733481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766935.733483:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766935.733487:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766935.733497:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766935.733503:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766935/real 1775766935] req@ffff9d81901aed80 x1862023066474752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766935.733510:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066474752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766935.733515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766935.733517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766935.733518:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766935.733522:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766935.733534:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766935.733540:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766935/real 1775766935] req@ffff9d81901afb80 x1862023066474880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766935.733547:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066474880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766935.733553:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766935.733555:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766935.733557:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066474880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766935.733561:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766935.733570:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766935.733575:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766935/real 1775766935] req@ffff9d81901af100 x1862023066475008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766935.733582:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066475008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766935.733587:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766935.733611:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766935.733613:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766936.373033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766936.756972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766937.397082:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766937.780955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766938.421099:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766938.804996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766939.445010:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766939.828983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766940.469085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766940.852993:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775766940.852994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766940.853002:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766940.853005:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766940.853013:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3772 00000100:00080000:3.0:1775766940.853021:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766940.853028:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766940.853058:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066475136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766940.853086:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766940.853089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766940.853091:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766940.853096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3772 00000100:00080000:3.0:1775766940.853099:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766940.853103:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766940.853113:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766940.853121:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066475136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766940.853125:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066475264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766940.853135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766940.853138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766940.853140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766940.853144:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3772 00000100:00080000:3.0:1775766940.853147:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766940.853150:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766940.853154:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766940.853163:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066475392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766940.853171:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766940.853174:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775766940.853175:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766940.853176:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766940.853179:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3772 00000100:00080000:3.0:1775766940.853183:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766940.853186:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766940.853190:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766940.853200:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066475520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766940.853208:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766940.853211:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766940.853213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766940.853216:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3772 00000100:00100000:0.0:1775766940.853217:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766940/real 1775766940] req@ffff9d8188743800 x1862023066475136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766940.853219:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766940.853223:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766940.853227:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766940.853235:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066475136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766940.853235:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066475648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766940.853247:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766940.853250:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766940.853254:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066475136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766940.853277:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766940.853298:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766940.853308:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766940/real 1775766940] req@ffff9d8188741f80 x1862023066475264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766940.853322:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066475264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766940.853331:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766940.853335:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766940.853338:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766940.853344:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766940.853364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766940.853375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766940/real 1775766940] req@ffff9d8188742300 x1862023066475392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766940.853388:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066475392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766940.853394:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766940.853396:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766940.853398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766940.853403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766940.853417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766940.853424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766940/real 1775766940] req@ffff9d8188740000 x1862023066475520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766940.853431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066475520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766940.853442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766940.853444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766940.853446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766940.853449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766940.853459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766940.853464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766940/real 1775766940] req@ffff9d8188740700 x1862023066475648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766940.853471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066475648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766940.853477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766940.853478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766940.853480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766941.493040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766941.876986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766942.517150:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766942.901008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766943.541042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766943.924941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766944.565133:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766944.948975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766945.589175:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766945.973008:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766945.973021:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775766945.973023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766945.973026:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766945.973039:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3777 00000100:00080000:3.0:1775766945.973054:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766945.973062:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766945.973119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066475776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766945.973164:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766945.973168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766945.973171:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766945.973174:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3777 00000100:00080000:3.0:1775766945.973178:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766945.973182:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766945.973193:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766945.973208:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066475904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766945.973220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766945.973225:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766945.973228:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766945.973233:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3777 00000100:00080000:3.0:1775766945.973240:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766945.973247:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766945.973254:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766945.973264:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066475776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766945.973276:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066476032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766945.973291:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766945.973297:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766945.973299:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766945.973306:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3777 00000100:00080000:3.0:1775766945.973312:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766945.973317:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766945.973324:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766945.973342:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066476160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766945.973352:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766945.973356:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766945.973360:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766945.973371:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766945.973379:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3777 00000100:00080000:3.0:1775766945.973388:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766945.973395:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766945.973404:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766945.973422:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066476288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766945.973441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766945/real 1775766945] req@ffff9d8188740380 x1862023066475776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766945.973475:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066475776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766945.973531:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766945.973540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766945.973549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066475776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766945.973566:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766945.973614:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766945.973634:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766945/real 1775766945] req@ffff9d8188741c00 x1862023066475904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766945.973667:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066475904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766945.973678:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766945.973682:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766945.973686:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066475904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766945.973717:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766945.973745:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766945.973758:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766945/real 1775766945] req@ffff9d8188742a00 x1862023066476032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766945.973773:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066476032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766945.973783:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766945.973787:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766945.973792:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766945.973800:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766945.973833:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766945.973854:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766945/real 1775766945] req@ffff9d8188740700 x1862023066476160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766945.973935:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066476160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766945.973956:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766945.973963:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766945.973970:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766945.973983:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766945.974028:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766945.974051:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766945/real 1775766945] req@ffff9d8188740000 x1862023066476288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766945.974068:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066476288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766945.974079:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766945.974084:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766945.974088:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766946.613082:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766946.996975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766947.637077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766948.021005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766948.661068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766949.044944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766949.685074:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766950.068990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766950.709002:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766951.092965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766951.092969:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766951.092976:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766951.092979:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766951.092986:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3782 00000100:00080000:3.0:1775766951.092995:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766951.093001:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766951.093034:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066476416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766951.093058:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766951.093062:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766951.093063:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766951.093066:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3782 00000100:00080000:3.0:1775766951.093070:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766951.093074:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766951.093083:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766951.093095:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066476544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766951.093105:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766951.093109:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766951.093110:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766951.093114:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3782 00000100:00100000:0.0:1775766951.093132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066476416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766951.093133:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766951.093137:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766951.093142:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766951.093153:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066476672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766951.093163:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766951.093167:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766951.093169:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766951.093173:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3782 00000100:00080000:3.0:1775766951.093184:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766951.093188:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775766951.093209:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775766951.093209:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766951.093221:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066476800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766951.093240:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766951.093244:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766951.093246:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766951.093250:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3782 00000100:00100000:0.0:1775766951.093253:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766951/real 1775766951] req@ffff9d8188742300 x1862023066476416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766951.093253:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766951.093257:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766951.093262:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766951.093272:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066476416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766951.093272:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066476928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766951.093283:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766951.093287:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766951.093306:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066476416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766951.093314:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766951.093336:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766951.093347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766951/real 1775766951] req@ffff9d8188741f80 x1862023066476544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766951.093356:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066476544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766951.093362:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766951.093364:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766951.093367:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766951.093384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766951.093396:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766951.093403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766951/real 1775766951] req@ffff9d8188743800 x1862023066476672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766951.093410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066476672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766951.093415:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766951.093417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766951.093419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766951.093423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766951.093447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766951.093454:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766951/real 1775766951] req@ffff9d8188742680 x1862023066476800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766951.093461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066476800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766951.093468:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766951.093470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766951.093472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766951.093475:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766951.093485:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766951.093490:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766951/real 1775766951] req@ffff9d8188741880 x1862023066476928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766951.093497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066476928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766951.093503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766951.093505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766951.093507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066476928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766951.733110:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766952.116979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766952.757016:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766953.140954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766953.781110:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766954.165022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766954.805062:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766955.188968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766955.829050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766956.212972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766956.212977:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766956.212983:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766956.212987:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766956.212994:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3787 00000100:00080000:3.0:1775766956.213004:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766956.213010:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766956.213061:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066477056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766956.213086:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766956.213089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766956.213091:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766956.213093:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3787 00000100:00080000:3.0:1775766956.213095:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766956.213107:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766956.213114:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766956.213122:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066477184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766956.213130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766956.213132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766956.213134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766956.213136:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3787 00000100:00080000:3.0:1775766956.213139:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766956.213141:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766956.213145:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766956.213149:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066477056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766956.213159:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066477312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766956.213178:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766956.213181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766956.213182:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766956.213185:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3787 00000100:00080000:3.0:1775766956.213188:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766956.213190:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766956.213194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766956.213202:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066477440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766956.213209:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766956.213211:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766956.213213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775766956.213217:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766956.213218:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3787 00000100:00080000:3.0:1775766956.213222:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766956.213225:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766956.213228:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766956.213237:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066477568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766956.213262:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766956/real 1775766956] req@ffff9d8188741180 x1862023066477056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766956.213287:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066477056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766956.213294:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766956.213297:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766956.213300:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066477056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766956.213306:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766956.213321:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766956.213329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766956/real 1775766956] req@ffff9d8188743100 x1862023066477184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766956.213337:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066477184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766956.213346:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766956.213348:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766956.213351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766956.213367:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766956.213404:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766956.213412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766956/real 1775766956] req@ffff9d8188740e00 x1862023066477312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766956.213419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066477312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766956.213424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766956.213426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766956.213428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766956.213433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766956.213444:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766956.213449:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766956/real 1775766956] req@ffff9d8188743480 x1862023066477440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766956.213457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066477440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766956.213462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766956.213464:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766956.213467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766956.213470:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766956.213480:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766956.213485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766956/real 1775766956] req@ffff9d8188743b80 x1862023066477568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766956.213498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066477568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766956.213504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766956.213506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766956.213508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766956.853132:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766957.237026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766957.877154:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766958.260971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766958.901042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766959.284952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766959.925121:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766960.308947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766960.949031:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766961.332933:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766961.332987:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766961.332993:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766961.332997:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766961.333004:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3792 00000100:00080000:1.0:1775766961.333012:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766961.333017:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766961.333052:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066477696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766961.333075:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766961.333077:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766961.333079:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766961.333081:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3792 00000100:00080000:1.0:1775766961.333084:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766961.333086:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766961.333094:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766961.333103:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066477824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766961.333111:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766961.333113:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766961.333114:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766961.333118:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3792 00000100:00080000:1.0:1775766961.333119:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766961.333123:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766961.333126:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066477696:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775766961.333127:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766961.333136:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066477952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766961.333143:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766961.333145:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766961.333146:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766961.333148:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3792 00000100:00080000:1.0:1775766961.333150:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766961.333153:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766961.333156:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766961.333164:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066478080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766961.333174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766961.333176:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766961.333178:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766961.333183:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3792 00000800:00000100:0.0:1775766961.333185:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775766961.333187:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766961.333190:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766961.333193:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766961.333202:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066478208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766961.333226:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766961/real 1775766961] req@ffff9d8190553480 x1862023066477696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766961.333241:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066477696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766961.333252:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766961.333257:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766961.333261:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066477696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766961.333270:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766961.333293:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766961.333306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766961/real 1775766961] req@ffff9d8190551880 x1862023066477824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766961.333318:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066477824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766961.333337:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766961.333340:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766961.333342:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766961.333361:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766961.333376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766961.333383:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766961/real 1775766961] req@ffff9d8190551500 x1862023066477952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766961.333390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066477952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766961.333396:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766961.333398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766961.333400:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066477952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766961.333403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766961.333413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766961.333418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766961/real 1775766961] req@ffff9d8190553100 x1862023066478080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766961.333426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066478080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766961.333431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766961.333433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766961.333435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766961.333438:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766961.333448:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766961.333453:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766961/real 1775766961] req@ffff9d8190550e00 x1862023066478208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766961.333460:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066478208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766961.333466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766961.333467:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766961.333469:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766961.973063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766962.356974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766962.997090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766963.380930:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766964.021172:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766964.404982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766965.045057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766965.428945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766966.069110:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766966.453040:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775766966.453056:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766966.453064:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775766966.453068:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766966.453077:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3797 00000100:00080000:0.0:1775766966.453086:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766966.453093:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766966.453139:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066478336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766966.453172:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766966.453176:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775766966.453178:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766966.453182:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3797 00000100:00080000:0.0:1775766966.453186:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766966.453190:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766966.453200:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766966.453214:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066478464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766966.453225:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766966.453229:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775766966.453230:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766966.453236:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3797 00000100:00080000:0.0:1775766966.453240:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775766966.453244:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066478336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775766966.453246:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766966.453256:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775766966.453286:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066478592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766966.453305:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766966.453308:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775766966.453310:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766966.453315:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3797 00000100:00080000:0.0:1775766966.453320:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766966.453327:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775766966.453327:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775766966.453336:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775766966.453349:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066478720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775766966.453356:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775766966.453359:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775766966.453360:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775766966.453364:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3797 00000100:00080000:0.0:1775766966.453367:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775766966.453371:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775766966.453375:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766966.453376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766966/real 1775766966] req@ffff9d81901af100 x1862023066478336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766966.453385:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066478848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766966.453397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066478336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766966.453406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766966.453410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766966.453414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066478336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766966.453423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766966.453445:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766966.453457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766966/real 1775766966] req@ffff9d81901afb80 x1862023066478464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766966.453470:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066478464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766966.453479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766966.453483:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766966.453487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766966.453507:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766966.453523:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766966.453530:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766966/real 1775766966] req@ffff9d81901aed80 x1862023066478592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766966.453537:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066478592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766966.453543:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766966.453545:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766966.453547:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766966.453551:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766966.453561:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766966.453567:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766966/real 1775766966] req@ffff9d81901ae680 x1862023066478720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766966.453574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066478720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766966.453580:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766966.453582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766966.453584:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766966.453587:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766966.453597:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766966.453602:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766966/real 1775766966] req@ffff9d81901ad180 x1862023066478848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766966.453609:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066478848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766966.453615:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766966.453616:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766966.453618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066478848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766967.093124:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766967.476990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766968.117126:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766968.501003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766969.141015:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766969.525034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766970.165061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766970.548970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766971.189183:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775766971.573109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766971.573118:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775766971.573122:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766971.573132:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3803 00000100:00080000:2.0:1775766971.573142:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766971.573152:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775766971.573169:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:2.0:1775766971.573185:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066478976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766971.573253:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766971.573257:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775766971.573259:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766971.573267:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3803 00000100:00080000:2.0:1775766971.573272:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766971.573277:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766971.573304:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766971.573321:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066479104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766971.573335:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766971.573341:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775766971.573343:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066478976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775766971.573344:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766971.573369:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3803 00000100:00080000:2.0:1775766971.573375:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766971.573382:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766971.573389:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775766971.573404:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066479232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775766971.573429:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775766971.573432:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775766971.573436:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775766971.573438:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766971.573442:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3803 00000100:00080000:2.0:1775766971.573447:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766971.573450:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766971.573455:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775766971.573465:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066479360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775766971.573472:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775766971.573475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766971/real 1775766971] req@ffff9d81984a2300 x1862023066478976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775766971.573492:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775766971.573496:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775766971.573504:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3803 00000100:00080000:2.0:1775766971.573510:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775766971.573517:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775766971.573524:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775766971.573543:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066479488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766971.573548:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066478976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766971.573572:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766971.573576:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766971.573580:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066478976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766971.573600:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766971.573636:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766971.573680:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766971/real 1775766971] req@ffff9d81984a2a00 x1862023066479104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766971.573694:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066479104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766971.573702:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766971.573732:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766971.573736:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766971.573742:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766971.573763:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766971.573773:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766971/real 1775766971] req@ffff9d81984a0700 x1862023066479232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766971.573804:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066479232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766971.573819:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766971.573822:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766971.573826:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766971.573831:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766971.573850:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766971.573886:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766971/real 1775766971] req@ffff9d81984a2d80 x1862023066479360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766971.573929:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066479360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766971.573938:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766971.573941:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766971.573944:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766971.573950:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766971.573969:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766971.573990:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766971/real 1775766971] req@ffff9d81984a3480 x1862023066479488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766971.574001:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066479488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775766996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766971.574009:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766971.574012:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766971.574015:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766972.213106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766972.596957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766973.237091:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766973.620986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766974.261057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766974.644961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766975.285035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766975.669026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766976.309063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766976.693017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766976.693023:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766976.693030:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766976.693034:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766976.693060:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3808 00000100:00080000:1.0:1775766976.693073:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766976.693084:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766976.693131:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066479616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766976.693189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766976.693193:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766976.693195:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766976.693218:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3808 00000100:00080000:1.0:1775766976.693222:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766976.693225:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766976.693232:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766976.693242:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066479744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766976.693251:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766976.693253:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766976.693255:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766976.693259:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3808 00000100:00080000:1.0:1775766976.693261:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766976.693263:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766976.693266:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766976.693273:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066479616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775766976.693274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066479872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766976.693286:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766976.693289:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766976.693290:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766976.693292:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3808 00000100:00080000:1.0:1775766976.693294:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766976.693296:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766976.693299:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766976.693308:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066480000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766976.693314:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766976.693316:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766976.693317:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766976.693319:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3808 00000100:00080000:1.0:1775766976.693324:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766976.693326:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766976.693329:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775766976.693334:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775766976.693338:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066480128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775766976.693375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766976/real 1775766976] req@ffff9d8190550e00 x1862023066479616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766976.693389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066479616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766976.693397:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766976.693399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766976.693402:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066479616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775766976.693407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766976.693420:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766976.693427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766976/real 1775766976] req@ffff9d8190553100 x1862023066479744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766976.693435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066479744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766976.693440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766976.693442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766976.693444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766976.693459:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775766976.693470:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766976.693475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766976/real 1775766976] req@ffff9d8190551500 x1862023066479872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766976.693483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066479872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766976.693488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766976.693489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766976.693491:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066479872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775766976.693495:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766976.693504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766976.693510:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766976/real 1775766976] req@ffff9d8190551880 x1862023066480000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766976.693517:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066480000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766976.693522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766976.693524:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766976.693526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775766976.693529:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775766976.693538:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766976.693543:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766976/real 1775766976] req@ffff9d8190553480 x1862023066480128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766976.693555:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066480128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766976.693561:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775766976.693563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766976.693564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766977.333043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766977.717005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766978.357136:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766978.741001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766979.381000:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766979.764986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766980.405040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766980.789012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766981.429185:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766981.813026:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766981.813035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766981.813035:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766981.813039:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766981.813053:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3813 00000100:00080000:3.0:1775766981.813063:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766981.813069:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766981.813119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066480256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766981.813154:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766981.813168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766981.813170:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766981.813175:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3813 00000100:00080000:3.0:1775766981.813179:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766981.813184:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766981.813194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766981.813205:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066480384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766981.813212:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066480256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766981.813231:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766981.813235:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766981.813237:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766981.813242:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3813 00000100:00080000:3.0:1775766981.813247:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766981.813252:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766981.813259:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766981.813276:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066480512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766981.813285:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766981.813288:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766981.813289:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766981.813292:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3813 00000100:00080000:3.0:1775766981.813305:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766981.813320:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766981.813324:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775766981.813335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775766981.813336:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066480640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766981.813345:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766981.813348:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766981.813350:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766981.813356:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3813 00000100:00080000:3.0:1775766981.813362:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766981.813365:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766981.813369:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775766981.813377:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066480768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766981.813405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766981/real 1775766981] req@ffff9d8188742d80 x1862023066480256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766981.813423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066480256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766981.813433:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766981.813436:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766981.813440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066480256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766981.813463:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766981.813481:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766981.813489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766981/real 1775766981] req@ffff9d8188740a80 x1862023066480384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766981.813497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066480384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766981.813503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766981.813505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766981.813507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766981.813511:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766981.813522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766981.813528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766981/real 1775766981] req@ffff9d8188741500 x1862023066480512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766981.813535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066480512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766981.813540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766981.813542:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766981.813544:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766981.813547:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766981.813558:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766981.813563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766981/real 1775766981] req@ffff9d8188743b80 x1862023066480640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766981.813571:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066480640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766981.813578:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766981.813581:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766981.813583:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766981.813586:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766981.813596:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766981.813601:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766981/real 1775766981] req@ffff9d8188743480 x1862023066480768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766981.813607:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066480768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766981.813613:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766981.813616:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766981.813618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066480768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766982.453051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766982.837024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766983.477160:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766983.861030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766984.501061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766984.884968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766985.525180:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766985.909049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766986.549103:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775766986.933036:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766986.933042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766986.933094:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766986.933099:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766986.933108:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3818 00000100:00080000:3.0:1775766986.933119:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766986.933126:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766986.933175:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066480896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766986.933201:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766986.933205:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766986.933206:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766986.933209:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3818 00000100:00080000:3.0:1775766986.933212:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766986.933223:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766986.933232:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766986.933242:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066481024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766986.933250:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066480896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766986.933268:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766986.933271:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775766986.933272:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766986.933277:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3818 00000100:00080000:3.0:1775766986.933279:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766986.933282:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766986.933287:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766986.933297:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066481152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766986.933305:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766986.933308:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775766986.933309:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766986.933312:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3818 00000100:00080000:3.0:1775766986.933315:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775766986.933320:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766986.933323:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766986.933328:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766986.933338:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066481280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766986.933345:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766986.933347:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766986.933349:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766986.933352:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3818 00000100:00080000:3.0:1775766986.933357:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766986.933360:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766986.933365:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775766986.933371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766986/real 1775766986] req@ffff9d8188740e00 x1862023066480896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775766986.933379:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066481408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766986.933413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066480896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766986.933428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766986.933434:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766986.933440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066480896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766986.933465:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766986.933488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766986.933499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766986/real 1775766986] req@ffff9d8188743100 x1862023066481024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766986.933510:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066481024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766986.933517:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766986.933519:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766986.933522:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766986.933526:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766986.933539:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766986.933546:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766986/real 1775766986] req@ffff9d8188741180 x1862023066481152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766986.933555:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066481152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766986.933567:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766986.933580:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766986.933582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766986.933586:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766986.933598:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766986.933621:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766986/real 1775766986] req@ffff9d8188741880 x1862023066481280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766986.933635:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066481280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766986.933647:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766986.933651:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766986.933654:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766986.933661:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766986.933689:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766986.933715:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766986/real 1775766986] req@ffff9d8188742680 x1862023066481408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766986.933725:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066481408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766986.933732:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766986.933735:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766986.933738:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766987.573106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766987.957011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766988.597093:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766988.981064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766989.621113:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775766990.004986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766990.645058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766991.029204:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766991.669094:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766992.052963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775766992.053051:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766992.053057:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775766992.053061:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766992.053068:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3823 00000100:00080000:1.0:1775766992.053077:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766992.053100:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775766992.053140:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066481536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766992.053169:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766992.053172:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775766992.053173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766992.053176:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3823 00000100:00080000:1.0:1775766992.053179:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766992.053181:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766992.053188:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766992.053200:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066481664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766992.053216:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066481536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775766992.053220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766992.053224:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775766992.053226:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766992.053231:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3823 00000100:00080000:1.0:1775766992.053247:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766992.053253:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766992.053258:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775766992.053273:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066481792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766992.053283:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766992.053287:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775766992.053289:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766992.053293:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3823 00000100:00080000:1.0:1775766992.053309:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775766992.053313:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766992.053319:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775766992.053332:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066481920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775766992.053343:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775766992.053347:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775766992.053349:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775766992.053354:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3823 00000100:00080000:1.0:1775766992.053360:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766992.053364:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481664:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775766992.053368:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775766992.053376:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775766992.053392:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066482048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766992.053397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766992.053419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766992.053436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766992.693054:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766993.076981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766993.717139:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766994.100999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775766994.741103:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766995.125137:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775766995.125437:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775766995.125528:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775766995.125551:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766995.127606:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766995.127624:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766995.127636:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775766995.127651:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775766995.127664:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766992/real 1775766995] req@ffff9d8190551880 x1862023066481664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766995.127700:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066481664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766995.127716:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766995.127722:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766995.127728:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766995.127738:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766992/real 1775766995] req@ffff9d8190551500 x1862023066481792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766995.127755:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066481792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766995.127767:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766995.127771:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766995.127775:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766995.127781:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766992/real 1775766995] req@ffff9d8190553100 x1862023066481920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766995.127791:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066481920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766995.127798:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766995.127801:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766995.127804:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066481920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766995.127809:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766992/real 1775766995] req@ffff9d8190553480 x1862023066481536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766995.127818:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066481536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766995.127823:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766995.127826:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766995.127828:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066481536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766995.127850:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766992/real 1775766995] req@ffff9d8190550e00 x1862023066482048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766995.127946:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066482048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766995.127983:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766995.127988:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766995.127993:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775766995.765183:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766996.149008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766996.789063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766997.172986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775766997.173064:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766997.173070:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775766997.173073:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766997.173081:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3828 00000100:00080000:3.0:1775766997.173090:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766997.173095:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775766997.173130:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066482176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766997.173162:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766997.173167:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775766997.173169:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766997.173173:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3828 00000100:00080000:3.0:1775766997.173177:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766997.173180:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766997.173189:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766997.173201:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066482304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766997.173211:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766997.173216:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775766997.173218:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066482176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775766997.173218:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766997.173225:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3828 00000100:00080000:3.0:1775766997.173231:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766997.173235:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766997.173241:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775766997.173297:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066482432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766997.173333:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775766997.173339:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775766997.173343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775766997.173344:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766997.173352:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3828 00000100:00080000:3.0:1775766997.173358:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775766997.173363:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766997.173369:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775766997.173386:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066482560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775766997.173398:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775766997.173399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766997/real 1775766997] req@ffff9d8188743800 x1862023066482176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766997.173409:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775766997.173414:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775766997.173424:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3828 00000100:00080000:3.0:1775766997.173429:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775766997.173436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066482176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775766997.173437:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775766997.173447:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775766997.173470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766997.173480:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775766997.173481:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066482688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775766997.173488:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066482176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775766997.173523:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766997.173571:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766997.173586:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766997/real 1775766997] req@ffff9d8188741f80 x1862023066482304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766997.173602:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066482304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766997.173612:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766997.173616:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766997.173619:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766997.173627:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775766997.173651:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766997.173663:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766997/real 1775766997] req@ffff9d8188742300 x1862023066482432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766997.173677:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066482432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766997.173686:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766997.173689:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766997.173692:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775766997.173698:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766997.173720:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766997.173731:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766997/real 1775766997] req@ffff9d8188740000 x1862023066482560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766997.173745:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066482560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766997.173756:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766997.173759:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766997.173764:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775766997.173771:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775766997.173792:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775766997.173803:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775766997/real 1775766997] req@ffff9d8188740700 x1862023066482688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775766997.173816:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066482688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775766997.173825:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775766997.173828:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775766997.173839:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775766997.813058:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766998.196995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766998.837036:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775766999.221055:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775766999.861260:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767000.245034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767000.885088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767001.269044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767001.909181:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767002.292982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767002.293031:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767002.293040:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767002.293046:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767002.293056:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3833 00000100:00080000:0.0:1775767002.293069:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767002.293076:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767002.293127:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066482816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767002.293157:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767002.293160:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767002.293162:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767002.293166:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3833 00000100:00080000:0.0:1775767002.293169:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767002.293172:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767002.293182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767002.293192:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066482944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767002.293200:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066482816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767002.293203:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767002.293209:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767002.293210:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767002.293219:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3833 00000100:00080000:0.0:1775767002.293221:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767002.293224:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767002.293228:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767002.293241:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066483072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767002.293248:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767002.293251:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767002.293252:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767002.293255:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3833 00000100:00080000:0.0:1775767002.293268:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767002.293282:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767002.293285:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767002.293292:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767002.293296:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066483200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767002.293303:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767002.293306:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767002.293308:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767002.293310:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3833 00000100:00080000:0.0:1775767002.293316:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767002.293318:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767002.293321:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767002.293329:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066483328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767002.293356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767002/real 1775767002] req@ffff9d81901ad180 x1862023066482816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767002.293390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066482816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767002.293404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767002.293409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767002.293414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066482816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767002.293445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767002.293472:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767002.293486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767002/real 1775767002] req@ffff9d81901ae680 x1862023066482944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767002.293498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066482944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767002.293506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767002.293510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767002.293514:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066482944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767002.293521:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767002.293543:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767002.293555:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767002/real 1775767002] req@ffff9d81901aed80 x1862023066483072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767002.293565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066483072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767002.293571:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767002.293574:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767002.293576:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767002.293581:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767002.293596:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767002.293603:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767002/real 1775767002] req@ffff9d81901afb80 x1862023066483200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767002.293616:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066483200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767002.293623:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767002.293625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767002.293627:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767002.293630:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767002.293641:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767002.293647:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767002/real 1775767002] req@ffff9d81901af100 x1862023066483328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767002.293654:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066483328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767002.293660:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767002.293661:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767002.293663:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767002.933094:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767003.317043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767003.957095:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767004.340984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767004.981052:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767005.364979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767006.005038:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767006.389006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767007.029121:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767007.413014:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767007.413028:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767007.413036:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767007.413036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767007.413053:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3838 00000100:00080000:0.0:1775767007.413064:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767007.413073:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767007.413113:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066483456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767007.413148:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767007.413154:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767007.413159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767007.413165:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3838 00000100:00080000:0.0:1775767007.413171:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767007.413177:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767007.413191:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767007.413214:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066483584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767007.413231:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767007.413235:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066483456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767007.413250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767007.413254:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767007.413265:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3838 00000100:00080000:0.0:1775767007.413272:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767007.413279:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767007.413288:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767007.413310:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066483712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767007.413328:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767007.413337:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775767007.413339:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767007.413343:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767007.413351:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3838 00000100:00080000:0.0:1775767007.413360:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767007.413365:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767007.413372:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767007.413389:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066483840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767007.413400:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767007.413407:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767007.413408:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767007/real 1775767007] req@ffff9d81901af100 x1862023066483456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767007.413414:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767007.413429:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3838 00000100:00080000:0.0:1775767007.413444:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767007.413451:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767007.413460:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767007.413466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066483456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767007.413476:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066483968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767007.413479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767007.413496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767007.413507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066483456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767007.413550:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767007.413584:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767007.413601:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767007/real 1775767007] req@ffff9d81901afb80 x1862023066483584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767007.413618:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066483584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767007.413628:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767007.413633:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767007.413637:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767007.413645:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767007.413668:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767007.413678:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767007/real 1775767007] req@ffff9d81901aed80 x1862023066483712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767007.413688:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066483712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767007.413695:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767007.413698:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767007.413701:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767007.413716:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767007.413733:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767007.413741:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767007/real 1775767007] req@ffff9d81901ae680 x1862023066483840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767007.413750:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066483840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767007.413758:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767007.413760:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767007.413763:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767007.413768:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767007.413779:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767007.413785:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767007/real 1775767007] req@ffff9d81901ad180 x1862023066483968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767007.413793:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066483968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767007.413800:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767007.413802:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767007.413804:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066483968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767008.053186:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767008.437017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767009.077026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767009.460989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767010.101147:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767010.485028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767011.125115:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767011.508966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767012.149017:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767012.532981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767012.532981:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767012.532994:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767012.533000:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767012.533011:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3844 00000100:00080000:2.0:1775767012.533023:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767012.533031:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767012.533091:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066484096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767012.533150:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767012.533153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767012.533155:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767012.533158:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3844 00000100:00080000:2.0:1775767012.533162:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767012.533165:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767012.533174:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767012.533187:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066484224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767012.533197:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767012.533201:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767012.533202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767012.533207:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066484096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767012.533210:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3844 00000100:00080000:2.0:1775767012.533218:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767012.533221:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767012.533225:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767012.533235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066484352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767012.533243:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767012.533246:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767012.533248:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767012.533251:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3844 00000100:00080000:2.0:1775767012.533264:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767012.533267:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767012.533271:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767012.533275:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767012.533288:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066484480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767012.533296:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767012.533299:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767012.533301:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767012.533304:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3844 00000100:00080000:2.0:1775767012.533308:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767012.533312:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767012.533316:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767012.533322:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767012/real 1775767012] req@ffff9d81984a2680 x1862023066484096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767012.533327:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066484608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767012.533370:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066484096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767012.533382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767012.533386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767012.533390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066484096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767012.533410:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767012.533429:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767012.533439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767012/real 1775767012] req@ffff9d81984a0e00 x1862023066484224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767012.533450:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066484224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767012.533457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767012.533459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767012.533462:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767012.533467:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767012.533479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767012.533486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767012/real 1775767012] req@ffff9d81984a1f80 x1862023066484352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767012.533501:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066484352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767012.533508:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767012.533510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767012.533512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767012.533516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767012.533549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767012.533558:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767012/real 1775767012] req@ffff9d81984a3b80 x1862023066484480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767012.533568:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066484480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767012.533576:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767012.533579:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767012.533582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767012.533587:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767012.533601:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767012.533608:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767012/real 1775767012] req@ffff9d81984a0000 x1862023066484608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767012.533618:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066484608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767012.533632:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767012.533635:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767012.533638:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767013.173139:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767013.556984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767014.197120:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767014.580986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767015.221127:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767015.605007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767016.245076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767016.629042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767017.269067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767017.652956:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767017.652964:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767017.652969:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767017.652978:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3849 00000100:00080000:2.0:1775767017.652989:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767017.652996:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767017.652999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:2.0:1775767017.653055:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066484736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767017.653105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767017.653109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767017.653111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767017.653114:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3849 00000100:00080000:2.0:1775767017.653117:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767017.653119:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767017.653127:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767017.653136:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066484864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767017.653144:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767017.653146:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767017.653147:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767017.653150:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3849 00000100:00080000:2.0:1775767017.653151:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767017.653153:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767017.653156:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767017.653161:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066484992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767017.653166:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767017.653168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767017.653170:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767017.653172:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3849 00000100:00080000:2.0:1775767017.653175:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767017.653179:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767017.653185:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767017.653196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066485120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767017.653202:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066484736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767017.653213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767017.653217:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767017.653220:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767017.653225:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3849 00000100:00080000:2.0:1775767017.653229:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767017.653233:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767017.653239:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767017.653251:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066485248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767017.653259:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767017.653299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767017/real 1775767017] req@ffff9d81984a1180 x1862023066484736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767017.653313:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066484736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767017.653321:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767017.653324:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767017.653327:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066484736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767017.653332:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767017.653345:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767017.653352:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767017/real 1775767017] req@ffff9d81984a1880 x1862023066484864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767017.653360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066484864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767017.653365:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767017.653368:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767017.653370:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767017.653374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767017.653383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767017.653388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767017/real 1775767017] req@ffff9d81984a3800 x1862023066484992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767017.653395:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066484992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767017.653401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767017.653403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767017.653405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066484992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767017.653423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767017.653433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767017.653439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767017/real 1775767017] req@ffff9d81984a3100 x1862023066485120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767017.653446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066485120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767017.653452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767017.653454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767017.653456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767017.653478:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767017.653495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767017.653503:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767017/real 1775767017] req@ffff9d81984a1500 x1862023066485248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767017.653515:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066485248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767017.653523:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767017.653526:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767017.653529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767018.293114:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767018.677019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767019.317057:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767019.700970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767020.341107:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767020.725042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767021.365095:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767021.748978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767022.389082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767022.772959:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767022.772961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767022.772969:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767022.772973:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767022.772982:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3854 00000100:00080000:2.0:1775767022.772991:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767022.772998:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767022.773033:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066485376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767022.773060:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767022.773063:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767022.773065:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767022.773067:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3854 00000100:00080000:2.0:1775767022.773071:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767022.773073:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767022.773081:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767022.773090:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066485504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767022.773098:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767022.773101:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767022.773102:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767022.773106:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3854 00000100:00080000:2.0:1775767022.773108:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767022.773110:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767022.773113:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767022.773121:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066485376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767022.773122:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066485632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767022.773136:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767022.773138:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767022.773139:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767022.773142:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3854 00000100:00080000:2.0:1775767022.773145:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767022.773147:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767022.773150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767022.773158:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066485760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767022.773170:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767022.773172:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767022.773173:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767022.773176:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3854 00000100:00080000:2.0:1775767022.773178:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767022.773180:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767022.773184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775767022.773188:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767022.773196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066485888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767022.773234:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767022/real 1775767022] req@ffff9d81984a0a80 x1862023066485376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767022.773255:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066485376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767022.773267:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767022.773271:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767022.773276:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066485376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767022.773283:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767022.773303:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767022.773314:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767022/real 1775767022] req@ffff9d81984a1500 x1862023066485504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767022.773328:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066485504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767022.773337:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767022.773341:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767022.773344:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767022.773365:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767022.773381:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767022.773389:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767022/real 1775767022] req@ffff9d81984a3100 x1862023066485632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767022.773402:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066485632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767022.773412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767022.773415:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767022.773419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767022.773424:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767022.773439:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767022.773448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767022/real 1775767022] req@ffff9d81984a3800 x1862023066485760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767022.773460:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066485760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767022.773478:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767022.773481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767022.773485:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767022.773491:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767022.773505:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767022.773513:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767022/real 1775767022] req@ffff9d81984a1880 x1862023066485888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767022.773525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066485888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767022.773534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767022.773538:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767022.773541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066485888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767023.413009:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767023.796956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767024.437066:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767024.821004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767025.461072:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767025.844993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767026.485063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767026.868969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767027.509051:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767027.892984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767027.893006:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767027.893013:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767027.893018:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767027.893025:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3859 00000100:00080000:1.0:1775767027.893033:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767027.893038:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767027.893070:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066486016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767027.893092:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767027.893095:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767027.893096:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767027.893099:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3859 00000100:00080000:1.0:1775767027.893101:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767027.893104:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767027.893111:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767027.893120:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066486144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767027.893127:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066486016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767027.893130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767027.893137:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767027.893139:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767027.893142:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3859 00000100:00080000:1.0:1775767027.893144:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767027.893147:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767027.893150:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767027.893159:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066486272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767027.893165:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767027.893168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767027.893169:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767027.893172:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3859 00000100:00080000:1.0:1775767027.893176:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767027.893178:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767027.893182:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767027.893187:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767027.893194:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066486400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767027.893201:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767027.893204:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767027.893205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767027.893207:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3859 00000100:00080000:1.0:1775767027.893210:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767027.893212:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767027.893217:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767027.893226:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066486528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767027.893230:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767027/real 1775767027] req@ffff9d8190550e00 x1862023066486016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767027.893251:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066486016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767027.893260:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767027.893263:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767027.893266:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066486016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767027.893283:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767027.893298:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767027.893306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767027/real 1775767027] req@ffff9d8190553480 x1862023066486144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767027.893314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066486144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767027.893320:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767027.893322:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767027.893324:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767027.893327:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767027.893336:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767027.893342:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767027/real 1775767027] req@ffff9d8190553100 x1862023066486272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767027.893349:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066486272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767027.893354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767027.893356:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767027.893358:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767027.893361:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767027.893371:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767027.893376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767027/real 1775767027] req@ffff9d8190551500 x1862023066486400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767027.893384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066486400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767027.893390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767027.893391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767027.893393:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767027.893396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767027.893406:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767027.893411:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767027/real 1775767027] req@ffff9d8190551880 x1862023066486528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767027.893418:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066486528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767027.893423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767027.893425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767027.893426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767028.533057:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767028.916958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767029.557009:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767029.940981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767030.581071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767030.964963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767031.605007:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767031.989064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767032.629069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767033.012942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767033.012967:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767033.012974:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767033.012979:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767033.012988:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3864 00000100:00080000:3.0:1775767033.012999:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767033.013006:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767033.013037:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066486656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767033.013061:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767033.013063:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767033.013065:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767033.013067:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3864 00000100:00080000:3.0:1775767033.013070:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767033.013073:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767033.013081:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767033.013093:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066486656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767033.013093:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066486784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767033.013108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767033.013110:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767033.013111:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767033.013114:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3864 00000100:00080000:3.0:1775767033.013117:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767033.013119:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767033.013122:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767033.013131:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066486912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767033.013136:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767033.013138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767033.013140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767033.013143:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3864 00000100:00080000:3.0:1775767033.013146:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767033.013148:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767033.013151:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767033.013154:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767033.013164:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066487040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767033.013171:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767033.013173:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767033.013174:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767033.013176:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3864 00000100:00080000:3.0:1775767033.013179:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767033.013181:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767033.013185:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767033.013194:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066487168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767033.013197:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767033/real 1775767033] req@ffff9d8188742a00 x1862023066486656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767033.013218:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066486656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767033.013225:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767033.013228:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767033.013233:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066486656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767033.013257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767033.013273:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767033.013281:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767033/real 1775767033] req@ffff9d8188741c00 x1862023066486784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767033.013289:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066486784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767033.013295:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767033.013297:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767033.013299:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767033.013303:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767033.013313:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767033.013318:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767033/real 1775767033] req@ffff9d8188740380 x1862023066486912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767033.013326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066486912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767033.013331:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767033.013333:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767033.013334:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066486912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767033.013338:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767033.013348:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767033.013354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767033/real 1775767033] req@ffff9d8188740700 x1862023066487040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767033.013366:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066487040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767033.013375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767033.013378:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767033.013381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767033.013388:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767033.013407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767033.013418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767033/real 1775767033] req@ffff9d8188740000 x1862023066487168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767033.013427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066487168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767033.013432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767033.013435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767033.013437:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767033.653053:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767034.036970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767034.677080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767035.060987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767035.701119:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767036.085008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767036.725021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767037.108999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767037.749017:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767038.132987:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767038.132990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767038.132997:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767038.133001:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767038.133008:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3869 00000100:00080000:3.0:1775767038.133017:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767038.133023:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767038.133058:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066487296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767038.133079:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767038.133082:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767038.133083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767038.133086:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3869 00000100:00080000:3.0:1775767038.133088:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767038.133090:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767038.133098:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767038.133107:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066487424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767038.133111:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066487296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767038.133123:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767038.133126:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767038.133127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767038.133130:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3869 00000100:00080000:3.0:1775767038.133132:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767038.133134:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767038.133137:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767038.133146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066487552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767038.133151:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767038.133153:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767038.133154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767038.133157:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3869 00000100:00080000:3.0:1775767038.133160:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767038.133163:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767038.133166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767038.133171:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767038.133178:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066487680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767038.133185:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767038.133187:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767038.133188:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767038.133190:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3869 00000100:00080000:3.0:1775767038.133192:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767038.133195:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767038.133198:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767038.133209:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066487808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767038.133213:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767038/real 1775767038] req@ffff9d8188742300 x1862023066487296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767038.133242:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066487296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767038.133251:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767038.133254:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767038.133257:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066487296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767038.133274:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767038.133288:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767038.133296:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767038/real 1775767038] req@ffff9d8188741f80 x1862023066487424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767038.133304:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066487424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767038.133310:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767038.133312:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767038.133314:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767038.133317:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767038.133327:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767038.133332:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767038/real 1775767038] req@ffff9d8188743800 x1862023066487552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767038.133344:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066487552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767038.133349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767038.133350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767038.133352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767038.133355:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767038.133366:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767038.133371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767038/real 1775767038] req@ffff9d8188742680 x1862023066487680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767038.133379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066487680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767038.133385:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767038.133386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767038.133388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767038.133391:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767038.133401:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767038.133406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767038/real 1775767038] req@ffff9d8188741880 x1862023066487808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767038.133421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066487808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767038.133431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767038.133433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767038.133435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066487808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767038.773097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767039.156999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767039.797081:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767040.181009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767040.821023:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767041.204969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767041.845093:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767042.228969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767042.869013:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767043.252975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767043.253035:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767043.253044:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767043.253048:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767043.253057:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3874 00000100:00080000:3.0:1775767043.253066:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767043.253071:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767043.253123:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066487936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767043.253147:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767043.253150:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767043.253151:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767043.253154:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3874 00000100:00080000:3.0:1775767043.253156:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767043.253158:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767043.253166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767043.253174:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066488064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767043.253182:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767043.253185:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767043.253186:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767043.253192:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3874 00000100:00080000:3.0:1775767043.253193:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767043.253196:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767043.253198:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767043.253206:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066487936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767043.253207:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066488192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767043.253228:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767043.253231:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767043.253232:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767043.253236:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3874 00000100:00080000:3.0:1775767043.253238:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767043.253240:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767043.253247:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767043.253260:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066488320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767043.253271:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767043.253274:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767043.253275:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767043.253279:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3874 00000100:00080000:3.0:1775767043.253292:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767043.253300:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767043.253301:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767043.253310:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767043.253326:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066488448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767043.253376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767043/real 1775767043] req@ffff9d8188741180 x1862023066487936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767043.253427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066487936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767043.253438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767043.253442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767043.253445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066487936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767043.253453:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767043.253496:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767043.253511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767043/real 1775767043] req@ffff9d8188743100 x1862023066488064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767043.253534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066488064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767043.253540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767043.253543:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767043.253545:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767043.253569:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767043.253586:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767043.253594:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767043/real 1775767043] req@ffff9d8188740e00 x1862023066488192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767043.253603:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066488192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767043.253609:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767043.253611:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767043.253613:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767043.253617:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767043.253629:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767043.253635:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767043/real 1775767043] req@ffff9d8188743480 x1862023066488320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767043.253642:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066488320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767043.253654:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767043.253656:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767043.253658:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767043.253662:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767043.253672:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767043.253678:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767043/real 1775767043] req@ffff9d8188743b80 x1862023066488448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767043.253685:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066488448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767043.253691:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767043.253693:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767043.253695:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767043.893046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767044.276974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767044.917018:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767045.300995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767045.941079:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767046.324977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767046.965091:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767047.348973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767047.989048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767048.372976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767048.373024:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767048.373032:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767048.373036:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767048.373046:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3879 00000100:00080000:1.0:1775767048.373057:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767048.373062:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767048.373096:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066488576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767048.373123:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767048.373126:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767048.373128:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767048.373130:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3879 00000100:00080000:1.0:1775767048.373133:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767048.373135:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767048.373143:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767048.373153:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066488704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767048.373163:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066488576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767048.373163:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767048.373179:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767048.373181:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767048.373185:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3879 00000100:00080000:1.0:1775767048.373188:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767048.373190:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767048.373195:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767048.373213:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066488832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767048.373219:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767048.373222:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767048.373224:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767048.373226:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3879 00000100:00080000:1.0:1775767048.373241:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767048.373244:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767048.373248:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767048.373255:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767048.373259:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066488960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767048.373267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767048.373270:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767048.373271:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767048.373273:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3879 00000100:00080000:1.0:1775767048.373276:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767048.373280:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767048.373282:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767048.373292:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066489088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767048.373326:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767048/real 1775767048] req@ffff9d8190551880 x1862023066488576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767048.373350:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066488576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767048.373359:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767048.373363:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767048.373367:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066488576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767048.373390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767048.373425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767048.373437:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767048/real 1775767048] req@ffff9d8190551500 x1862023066488704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767048.373447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066488704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767048.373453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767048.373456:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767048.373458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767048.373463:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767048.373477:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767048.373483:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767048/real 1775767048] req@ffff9d8190553100 x1862023066488832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767048.373491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066488832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767048.373501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767048.373503:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767048.373506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767048.373509:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767048.373520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767048.373526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767048/real 1775767048] req@ffff9d8190553480 x1862023066488960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767048.373533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066488960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767048.373540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767048.373542:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767048.373544:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066488960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767048.373548:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767048.373558:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767048.373563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767048/real 1775767048] req@ffff9d8190550e00 x1862023066489088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767048.373571:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066489088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767048.373576:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767048.373578:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767048.373581:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767049.013146:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767049.396995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767050.037117:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767050.420994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767051.061068:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767051.444988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767052.085025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767052.468946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767053.109084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767053.493032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767053.493208:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767053.493213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767053.493217:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767053.493225:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3885 00000100:00080000:3.0:1775767053.493233:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767053.493238:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767053.493272:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066489216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767053.493295:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767053.493298:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767053.493299:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767053.493302:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3885 00000100:00080000:3.0:1775767053.493304:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767053.493306:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767053.493313:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767053.493321:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066489344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767053.493333:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767053.493337:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066489216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767053.493339:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767053.493343:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767053.493349:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3885 00000100:00080000:3.0:1775767053.493353:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767053.493369:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767053.493375:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767053.493391:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066489472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767053.493412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767053.493421:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767053.493425:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767053.493427:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767053.493431:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3885 00000100:00080000:3.0:1775767053.493435:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767053.493438:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767053.493442:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767053.493455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767053/real 1775767053] req@ffff9d8188741500 x1862023066489216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767053.493456:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066489600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767053.493488:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767053.493490:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767053.493491:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767053.493494:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3885 00000100:00100000:0.0:1775767053.493497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066489216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767053.493498:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767053.493504:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767053.493511:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767053.493516:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767053.493521:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066489728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767053.493522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767053.493529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066489216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767053.493554:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767053.493570:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767053.493577:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767053/real 1775767053] req@ffff9d8188740a80 x1862023066489344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767053.493586:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066489344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767053.493592:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767053.493594:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767053.493597:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767053.493600:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767053.493611:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767053.493617:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767053/real 1775767053] req@ffff9d8188742d80 x1862023066489472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767053.493625:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066489472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767053.493630:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767053.493632:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767053.493634:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767053.493637:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767053.493647:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767053.493652:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767053/real 1775767053] req@ffff9d8188743b80 x1862023066489600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767053.493659:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066489600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767053.493665:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767053.493667:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767053.493669:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767053.493672:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767053.493691:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767053.493697:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767053/real 1775767053] req@ffff9d8188743480 x1862023066489728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767053.493705:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066489728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767053.493710:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767053.493713:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767053.493715:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767054.133062:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767054.516976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767055.157025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767055.541002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767056.181094:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767056.565008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767057.205021:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767057.589000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767058.229025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767058.612969:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767058.612973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767058.612982:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767058.612987:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767058.612997:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3890 00000100:00080000:3.0:1775767058.613008:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767058.613016:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767058.613052:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066489856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767058.613081:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767058.613086:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767058.613088:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767058.613092:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3890 00000100:00080000:3.0:1775767058.613096:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767058.613101:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767058.613111:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767058.613115:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066489856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767058.613128:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066489984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767058.613140:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767058.613143:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767058.613145:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767058.613151:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3890 00000100:00080000:3.0:1775767058.613164:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767058.613169:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767058.613174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767058.613186:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066490112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767058.613203:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767058.613207:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767058.613209:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767058.613214:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3890 00000100:00080000:3.0:1775767058.613218:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767058.613222:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489984:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775767058.613223:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767058.613233:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767058.613248:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066490240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767058.613259:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767058.613261:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490112:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775767058.613267:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767058.613270:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767058.613274:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3890 00000100:00080000:3.0:1775767058.613279:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767058.613283:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767058.613289:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767058.613294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767058.613320:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066490368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767058.613342:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767059.253046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767059.636983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767060.277073:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767060.660948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767061.301080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767061.684999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767061.685120:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767061.685151:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767061.685164:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767061.685207:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767061.685219:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767061.685229:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767061.685240:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767061.685246:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767058/real 1775767061] req@ffff9d8188743100 x1862023066489984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767061.685270:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066489984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767061.685293:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767061.685299:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767061.685304:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066489984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767061.685316:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767058/real 1775767061] req@ffff9d8188741180 x1862023066490112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767061.685332:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066490112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767061.685340:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767061.685344:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767061.685347:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767061.685354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767058/real 1775767061] req@ffff9d8188740e00 x1862023066489856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767061.685372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066489856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767061.685391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767061.685395:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767061.685399:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066489856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767061.685424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767058/real 1775767061] req@ffff9d8188742680 x1862023066490368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767061.685437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066490368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767061.685445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767061.685448:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767061.685451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490368:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767061.685458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767058/real 1775767061] req@ffff9d8188741880 x1862023066490240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767061.685467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066490240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767061.685476:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767061.685480:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767061.685482:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490240:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767062.325031:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767062.709005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767063.349030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767063.732995:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767063.733004:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767063.733008:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767063.733016:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3895 00000100:00080000:3.0:1775767063.733026:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767063.733031:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767063.733046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775767063.733065:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066490496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767063.733093:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767063.733096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767063.733097:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767063.733099:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3895 00000100:00080000:3.0:1775767063.733101:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767063.733104:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767063.733112:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767063.733124:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066490496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767063.733124:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066490624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767063.733138:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767063.733140:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767063.733141:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767063.733144:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3895 00000100:00080000:3.0:1775767063.733146:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767063.733148:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767063.733151:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767063.733160:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066490752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767063.733174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767063.733178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767063.733179:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767063.733183:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3895 00000100:00080000:3.0:1775767063.733187:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767063.733191:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767063.733197:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767063.733198:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767063.733215:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066490880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767063.733226:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767063.733229:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767063.733231:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767063.733235:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3895 00000100:00080000:3.0:1775767063.733240:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767063.733244:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767063.733266:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767063.733278:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066491008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767063.733288:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767063/real 1775767063] req@ffff9d8188743800 x1862023066490496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767063.733317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066490496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767063.733327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767063.733332:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767063.733337:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066490496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767063.733358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767063.733383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767063.733398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767063/real 1775767063] req@ffff9d8188741f80 x1862023066490624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767063.733410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066490624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767063.733418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767063.733421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767063.733424:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767063.733430:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767063.733451:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767063.733462:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767063/real 1775767063] req@ffff9d8188742300 x1862023066490752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767063.733474:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066490752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767063.733486:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767063.733490:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767063.733493:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767063.733498:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767063.733518:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767063.733527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767063/real 1775767063] req@ffff9d8188740000 x1862023066490880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767063.733538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066490880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767063.733547:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767063.733550:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767063.733553:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066490880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767063.733560:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767063.733580:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767063.733590:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767063/real 1775767063] req@ffff9d8188740700 x1862023066491008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767063.733602:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066491008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767063.733611:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767063.733615:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767063.733618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767064.373034:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767064.757054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767065.397080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767065.780979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767066.420982:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767066.804955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767067.445041:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767067.828948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767068.469068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767068.852979:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767068.852983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767068.852991:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767068.852994:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767068.853003:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3900 00000100:00080000:3.0:1775767068.853012:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767068.853018:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767068.853052:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066491136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767068.853074:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767068.853077:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767068.853078:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767068.853081:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3900 00000100:00080000:3.0:1775767068.853083:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767068.853086:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767068.853094:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767068.853105:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066491264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767068.853106:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066491136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767068.853120:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767068.853123:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767068.853124:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767068.853127:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3900 00000100:00080000:3.0:1775767068.853129:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767068.853131:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767068.853134:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767068.853143:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066491392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767068.853150:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767068.853152:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767068.853153:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767068.853156:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3900 00000100:00080000:3.0:1775767068.853159:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767068.853160:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767068.853163:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767068.853168:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767068.853186:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066491520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767068.853192:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767068.853195:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767068.853211:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767068.853214:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3900 00000100:00080000:3.0:1775767068.853217:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767068.853221:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767068.853225:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767068.853233:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767068/real 1775767068] req@ffff9d8188740380 x1862023066491136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767068.853238:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066491648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767068.853258:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066491136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767068.853267:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767068.853271:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767068.853275:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066491136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767068.853291:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767068.853306:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767068.853314:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767068/real 1775767068] req@ffff9d8188741c00 x1862023066491264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767068.853322:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066491264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767068.853327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767068.853330:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767068.853332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767068.853336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767068.853346:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767068.853351:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767068/real 1775767068] req@ffff9d8188742a00 x1862023066491392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767068.853359:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066491392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767068.853363:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767068.853365:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767068.853367:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767068.853370:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767068.853380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767068.853385:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767068/real 1775767068] req@ffff9d8188740700 x1862023066491520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767068.853392:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066491520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767068.853398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767068.853399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767068.853401:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767068.853404:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767068.853414:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767068.853419:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767068/real 1775767068] req@ffff9d8188740000 x1862023066491648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767068.853426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066491648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767068.853431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767068.853433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767068.853434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767069.493029:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767069.876952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767070.517027:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767070.901031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767071.541102:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767071.924963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767072.565071:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767072.949040:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767073.589117:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767073.972948:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767073.972949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767073.972958:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767073.972964:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767073.972975:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3905 00000100:00080000:3.0:1775767073.972986:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767073.972993:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767073.973035:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066491776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767073.973092:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767073.973096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767073.973098:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767073.973102:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3905 00000100:00080000:3.0:1775767073.973122:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767073.973125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767073.973134:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767073.973155:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066491904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767073.973164:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767073.973167:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767073.973169:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767073.973173:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3905 00000100:00080000:3.0:1775767073.973176:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767073.973178:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767073.973183:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767073.973185:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066491776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767073.973194:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066492032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767073.973211:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767073.973215:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767073.973217:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767073.973222:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3905 00000100:00080000:3.0:1775767073.973226:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767073.973230:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767073.973236:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767073.973252:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066492160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767073.973262:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767073.973265:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767073.973270:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767073.973273:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767073.973279:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3905 00000100:00080000:3.0:1775767073.973287:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767073.973292:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767073.973298:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767073.973313:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066492288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767073.973318:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767073/real 1775767073] req@ffff9d8188742300 x1862023066491776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767073.973350:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066491776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767073.973359:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767073.973363:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767073.973366:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066491776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767073.973389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767073.973412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767073.973422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767073/real 1775767073] req@ffff9d8188741f80 x1862023066491904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767073.973432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066491904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767073.973439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767073.973442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767073.973445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066491904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767073.973468:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767073.973481:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767073.973488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767073/real 1775767073] req@ffff9d8188743800 x1862023066492032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767073.973497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066492032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767073.973504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767073.973506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767073.973508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767073.973513:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767073.973525:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767073.973533:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767073/real 1775767073] req@ffff9d8188741880 x1862023066492160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767073.973542:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066492160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767073.973549:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767073.973552:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767073.973554:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767073.973558:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767073.973569:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767073.973576:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767073/real 1775767073] req@ffff9d8188742680 x1862023066492288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767073.973584:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066492288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767073.973591:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767073.973594:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767073.973598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767074.613024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767074.997024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767075.637055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767076.020966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767076.661059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767077.044983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767077.685032:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767078.069024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767078.709056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767079.093000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767079.093018:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767079.093024:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767079.093027:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767079.093035:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3910 00000100:00080000:2.0:1775767079.093051:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767079.093058:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767079.093131:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066492416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767079.093154:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767079.093156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767079.093157:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767079.093160:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3910 00000100:00080000:2.0:1775767079.093163:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767079.093167:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767079.093175:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767079.093189:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066492416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767079.093189:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066492544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767079.093203:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767079.093206:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767079.093207:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767079.093210:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3910 00000100:00080000:2.0:1775767079.093212:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767079.093214:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767079.093217:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767079.093227:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066492672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767079.093233:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767079.093236:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767079.093237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767079.093240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3910 00000100:00080000:2.0:1775767079.093243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767079.093245:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767079.093248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767079.093252:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767079.093260:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066492800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767079.093267:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767079.093269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767079.093271:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767079.093274:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3910 00000100:00080000:2.0:1775767079.093276:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767079.093278:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767079.093281:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767079.093295:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066492928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767079.093305:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767079/real 1775767079] req@ffff9d81984a1180 x1862023066492416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767079.093325:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066492416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767079.093335:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767079.093339:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767079.093343:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066492416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767079.093366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767079.093390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767079.093403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767079/real 1775767079] req@ffff9d81984a0000 x1862023066492544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767079.093431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066492544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767079.093440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767079.093444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767079.093447:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767079.093453:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767079.093475:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767079.093486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767079/real 1775767079] req@ffff9d81984a3b80 x1862023066492672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767079.093507:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066492672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767079.093528:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767079.093531:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767079.093534:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767079.093540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767079.093560:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767079.093571:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767079/real 1775767079] req@ffff9d81984a1f80 x1862023066492800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767079.093583:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066492800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767079.093593:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767079.093597:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767079.093599:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767079.093618:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767079.093638:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767079.093658:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767079/real 1775767079] req@ffff9d81984a0e00 x1862023066492928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767079.093670:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066492928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767079.093697:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767079.093700:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767079.093704:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066492928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767079.733039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767080.116982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767080.757029:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767081.141000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767081.781057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767082.164957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767082.805092:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767083.189024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767083.829100:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767084.212965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767084.213009:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767084.213015:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767084.213019:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767084.213028:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3915 00000100:00080000:2.0:1775767084.213051:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767084.213075:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767084.213111:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066493056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767084.213135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767084.213138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767084.213140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767084.213142:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3915 00000100:00080000:2.0:1775767084.213145:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767084.213148:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767084.213156:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767084.213165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066493184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767084.213170:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066493056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767084.213194:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767084.213199:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767084.213202:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767084.213208:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3915 00000100:00080000:2.0:1775767084.213220:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767084.213225:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767084.213231:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767084.213234:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767084.213251:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066493312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767084.213279:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767084.213283:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767084.213285:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767084.213289:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3915 00000100:00080000:2.0:1775767084.213294:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767084.213299:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767084.213300:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767084/real 1775767084] req@ffff9d81984a2680 x1862023066493056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767084.213305:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767084.213322:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066493440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767084.213332:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066493056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767084.213337:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767084.213347:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767084.213351:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767084.213357:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3915 00000100:00080000:2.0:1775767084.213362:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767084.213363:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767084.213369:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775767084.213371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767084.213378:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066493056:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775767084.213380:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767084.213395:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066493568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767084.213403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767084.213427:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767084.213441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767084/real 1775767084] req@ffff9d81984a3480 x1862023066493184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767084.213456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066493184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767084.213462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767084.213465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767084.213467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767084.213471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767084.213483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767084.213490:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767084/real 1775767084] req@ffff9d81984a2d80 x1862023066493312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767084.213498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066493312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767084.213504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767084.213506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767084.213507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767084.213528:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767084.213540:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767084.213546:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767084/real 1775767084] req@ffff9d81984a0700 x1862023066493440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767084.213553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066493440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767084.213564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767084.213566:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767084.213568:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767084.213594:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767084.213606:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767084.213612:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767084/real 1775767084] req@ffff9d81984a2a00 x1862023066493568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767084.213619:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066493568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767084.213625:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767084.213627:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767084.213629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767084.853072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767085.237021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767085.877079:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767086.260993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767086.901084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767087.285025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767087.925093:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767088.309078:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767088.949106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767089.332956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767089.332960:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767089.332968:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767089.332973:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767089.332998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3920 00000100:00080000:2.0:1775767089.333009:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767089.333017:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767089.333061:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066493696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767089.333091:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767089.333095:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767089.333097:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767089.333102:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3920 00000100:00080000:2.0:1775767089.333107:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767089.333113:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767089.333120:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066493696:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775767089.333123:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767089.333143:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066493824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767089.333155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767089.333162:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767089.333164:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767089.333170:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3920 00000100:00080000:2.0:1775767089.333174:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767089.333181:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767089.333182:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767089.333190:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767089.333205:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066493952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767089.333217:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767089.333221:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767089.333223:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767089.333228:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3920 00000100:00100000:0.0:1775767089.333232:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767089/real 1775767089] req@ffff9d81984a2300 x1862023066493696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767089.333235:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767089.333245:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767089.333256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767089.333264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066493696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767089.333272:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066494080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767089.333275:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767089.333283:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767089.333289:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066493696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767089.333295:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767089.333299:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767089.333302:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767089.333307:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3920 00000100:00100000:0.0:1775767089.333311:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493824:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775767089.333313:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767089.333319:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767089.333327:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767089.333334:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767089.333342:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066494208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767089.333346:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767089/real 1775767089] req@ffff9d81984a2a00 x1862023066493824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767089.333374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066493824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767089.333380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767089.333383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767089.333386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767089.333391:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767089.333402:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767089.333408:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767089/real 1775767089] req@ffff9d81984a0700 x1862023066493952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767089.333416:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066493952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767089.333425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767089.333427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767089.333429:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066493952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767089.333443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767089.333455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767089.333461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767089/real 1775767089] req@ffff9d81984a2d80 x1862023066494080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767089.333468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066494080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767089.333474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767089.333476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767089.333478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767089.333487:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767089.333497:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767089.333502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767089/real 1775767089] req@ffff9d81984a3480 x1862023066494208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767089.333509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066494208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767089.333515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767089.333516:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767089.333519:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767089.973126:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767090.357076:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767090.997164:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767091.380942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767092.021106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767092.405134:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767093.045073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767093.428965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767094.069073:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767094.452997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767094.452999:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767094.453006:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767094.453010:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767094.453024:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3925 00000100:00080000:2.0:1775767094.453033:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767094.453040:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767094.453073:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066494336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767094.453096:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767094.453098:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767094.453100:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767094.453102:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3925 00000100:00080000:2.0:1775767094.453105:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767094.453107:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767094.453114:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767094.453123:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066494464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767094.453131:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767094.453135:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767094.453136:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066494336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767094.453137:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767094.453144:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3925 00000100:00080000:2.0:1775767094.453148:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767094.453151:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767094.453153:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767094.453164:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066494592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767094.453171:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767094.453173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767094.453174:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767094.453177:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3925 00000100:00080000:2.0:1775767094.453189:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767094.453192:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767094.453196:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767094.453200:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767094.453209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066494720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767094.453217:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767094.453220:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767094.453221:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767094.453223:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3925 00000100:00080000:2.0:1775767094.453226:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767094.453228:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767094.453231:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767094.453240:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066494848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767094.453256:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767094/real 1775767094] req@ffff9d81984a2680 x1862023066494336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767094.453279:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066494336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767094.453291:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767094.453296:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767094.453300:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066494336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767094.453323:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767094.453342:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767094.453351:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767094/real 1775767094] req@ffff9d81984a0e00 x1862023066494464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767094.453360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066494464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767094.453366:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767094.453368:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767094.453370:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767094.453374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767094.453385:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767094.453391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767094/real 1775767094] req@ffff9d81984a1f80 x1862023066494592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767094.453398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066494592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767094.453403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767094.453405:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767094.453407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767094.453411:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767094.453421:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767094.453426:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767094/real 1775767094] req@ffff9d81984a3b80 x1862023066494720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767094.453434:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066494720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767094.453439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767094.453441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767094.453443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767094.453446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767094.453456:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767094.453461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767094/real 1775767094] req@ffff9d81984a0000 x1862023066494848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767094.453468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066494848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767094.453473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767094.453475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767094.453477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066494848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767095.093030:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767095.476993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767096.117055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767096.500993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767097.141079:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767097.524966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767098.165122:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767098.549027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767099.189053:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767099.572949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767099.572964:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767099.572970:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767099.572974:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767099.572982:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3931 00000100:00080000:2.0:1775767099.572990:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767099.572996:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767099.573029:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066494976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767099.573052:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767099.573054:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767099.573056:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767099.573058:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3931 00000100:00080000:2.0:1775767099.573060:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767099.573063:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767099.573071:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767099.573083:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066494976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767099.573083:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066495104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767099.573096:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767099.573099:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767099.573100:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767099.573103:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3931 00000100:00080000:2.0:1775767099.573105:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767099.573107:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767099.573111:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767099.573121:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066495232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767099.573127:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767099.573130:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767099.573131:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767099.573134:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3931 00000100:00080000:2.0:1775767099.573136:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767099.573138:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767099.573141:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767099.573143:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767099.573151:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066495360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767099.573159:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767099.573162:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767099.573163:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767099.573166:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3931 00000100:00080000:2.0:1775767099.573168:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767099.573171:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767099.573174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767099.573183:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066495488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767099.573188:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767099/real 1775767099] req@ffff9d81984a1180 x1862023066494976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767099.573207:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066494976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767099.573215:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767099.573219:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767099.573222:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066494976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767099.573242:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767099.573257:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767099.573264:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767099/real 1775767099] req@ffff9d81984a1880 x1862023066495104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767099.573273:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066495104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767099.573278:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767099.573281:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767099.573283:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767099.573286:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767099.573296:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767099.573301:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767099/real 1775767099] req@ffff9d81984a3800 x1862023066495232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767099.573308:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066495232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767099.573313:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767099.573315:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767099.573317:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767099.573320:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767099.573330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767099.573336:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767099/real 1775767099] req@ffff9d81984a3100 x1862023066495360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767099.573343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066495360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767099.573348:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767099.573350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767099.573352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767099.573355:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767099.573365:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767099.573370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767099/real 1775767099] req@ffff9d81984a1500 x1862023066495488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767099.573377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066495488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767099.573382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767099.573384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767099.573385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767100.213046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767100.597019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767101.237061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767101.620936:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767102.261066:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767102.645003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767103.285038:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767103.668965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767104.309062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767104.692972:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767104.692981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767104.692982:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767104.692987:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767104.692996:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3936 00000100:00080000:0.0:1775767104.693005:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767104.693011:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767104.693045:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066495616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767104.693080:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767104.693083:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767104.693085:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767104.693089:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3936 00000100:00080000:0.0:1775767104.693093:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767104.693097:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767104.693106:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767104.693117:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066495744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767104.693118:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066495616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767104.693128:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767104.693131:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767104.693133:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767104.693136:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3936 00000100:00080000:0.0:1775767104.693140:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767104.693143:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767104.693147:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767104.693155:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066495872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767104.693163:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767104.693165:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1775767104.693173:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775767104.693176:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767104.693180:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3936 00000100:00080000:0.0:1775767104.693186:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767104.693190:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767104.693194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767104.693203:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066496000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767104.693212:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767104.693216:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1775767104.693216:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767104/real 1775767104] req@ffff9d81901ad180 x1862023066495616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767104.693217:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767104.693221:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3936 00000100:00080000:0.0:1775767104.693224:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767104.693228:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767104.693232:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767104.693234:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066495616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767104.693240:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066496128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767104.693244:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.693248:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767104.693252:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066495616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767104.693282:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767104.693305:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767104.693315:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767104/real 1775767104] req@ffff9d81901ae680 x1862023066495744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.693325:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066495744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.693331:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.693333:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767104.693335:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767104.693339:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767104.693351:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767104.693356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767104/real 1775767104] req@ffff9d81901aed80 x1862023066495872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.693364:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066495872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.693369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.693370:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767104.693372:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066495872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767104.693376:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767104.693386:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767104.693391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767104/real 1775767104] req@ffff9d81901afb80 x1862023066496000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.693398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066496000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.693408:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.693410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767104.693412:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767104.693415:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767104.693424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767104.693429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767104/real 1775767104] req@ffff9d81901af100 x1862023066496128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767104.693436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066496128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767104.693441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767104.693443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767104.693444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767105.333022:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767105.717011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767106.357046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767106.740956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767107.381042:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767107.764954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767108.405082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767108.788956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767109.429022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767109.812973:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767109.812982:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775767109.812983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767109.812986:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767109.812995:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3941 00000100:00080000:0.0:1775767109.813005:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767109.813011:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767109.813044:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066496256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767109.813068:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767109.813071:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767109.813072:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767109.813075:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3941 00000100:00080000:0.0:1775767109.813081:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767109.813084:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767109.813091:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767109.813099:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066496384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767109.813107:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767109.813109:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767109.813110:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767109.813113:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3941 00000100:00080000:0.0:1775767109.813115:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767109.813117:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066496256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767109.813120:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767109.813127:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767109.813135:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066496512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767109.813147:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767109.813150:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767109.813151:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767109.813153:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3941 00000100:00080000:0.0:1775767109.813164:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767109.813167:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767109.813171:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767109.813175:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767109.813182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066496640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767109.813189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767109.813197:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767109.813199:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767109.813201:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3941 00000100:00080000:0.0:1775767109.813203:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767109.813207:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767109.813210:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767109.813217:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767109/real 1775767109] req@ffff9d81901af100 x1862023066496256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767109.813220:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066496768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767109.813263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066496256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767109.813282:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767109.813286:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767109.813289:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066496256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767109.813295:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767109.813313:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767109.813322:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767109/real 1775767109] req@ffff9d81901afb80 x1862023066496384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767109.813330:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066496384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767109.813336:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767109.813338:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767109.813340:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767109.813356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767109.813367:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767109.813373:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767109/real 1775767109] req@ffff9d81901aed80 x1862023066496512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767109.813380:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066496512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767109.813386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767109.813387:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767109.813390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767109.813394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767109.813404:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767109.813409:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767109/real 1775767109] req@ffff9d81901ae680 x1862023066496640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767109.813424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066496640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767109.813431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767109.813433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767109.813435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767109.813439:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767109.813450:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767109.813455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767109/real 1775767109] req@ffff9d81901ad180 x1862023066496768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767109.813463:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066496768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767109.813468:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767109.813470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767109.813472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066496768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767110.453057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767110.836959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767111.477098:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767111.861011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767112.501097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767112.885029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767113.525063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767113.909009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767114.549123:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767114.933001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767114.933011:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767114.933020:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767114.933024:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767114.933033:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3946 00000100:00080000:3.0:1775767114.933043:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767114.933049:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767114.933082:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066496896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767114.933108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767114.933112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767114.933114:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767114.933117:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3946 00000100:00080000:3.0:1775767114.933121:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767114.933125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767114.933134:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767114.933146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066497024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767114.933148:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066496896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767114.933157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767114.933160:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767114.933162:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767114.933166:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3946 00000100:00080000:3.0:1775767114.933169:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767114.933172:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767114.933177:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767114.933186:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066497152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767114.933194:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767114.933197:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767114.933199:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767114.933202:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3946 00000100:00080000:3.0:1775767114.933205:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767114.933208:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767114.933213:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767114.933213:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767114.933221:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066497280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767114.933229:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767114.933232:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767114.933233:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767114.933236:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3946 00000100:00080000:3.0:1775767114.933239:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767114.933242:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767114.933246:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767114.933254:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066497408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767114.933261:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767114/real 1775767114] req@ffff9d8188740e00 x1862023066496896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767114.933280:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066496896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767114.933289:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767114.933294:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767114.933297:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066496896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767114.933318:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767114.933336:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767114.933344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767114/real 1775767114] req@ffff9d8188741180 x1862023066497024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767114.933353:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066497024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767114.933359:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767114.933362:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767114.933366:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767114.933371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767114.933390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767114.933401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767114/real 1775767114] req@ffff9d8188743100 x1862023066497152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767114.933414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066497152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767114.933429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767114.933432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767114.933434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767114.933438:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767114.933453:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767114.933460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767114/real 1775767114] req@ffff9d8188743480 x1862023066497280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767114.933468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066497280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767114.933474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767114.933476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767114.933477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767114.933481:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767114.933491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767114.933496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767114/real 1775767114] req@ffff9d8188743b80 x1862023066497408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767114.933503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066497408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767114.933508:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767114.933510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767114.933512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767115.573017:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767115.957018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767116.597081:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767116.981005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767117.621146:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767118.005014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767118.645109:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767119.028991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767119.669059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767120.052970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767120.052972:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767120.052980:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767120.052984:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767120.052991:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3951 00000100:00080000:3.0:1775767120.053001:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767120.053006:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767120.053039:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066497536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767120.053064:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767120.053066:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767120.053068:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767120.053070:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3951 00000100:00080000:3.0:1775767120.053072:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767120.053074:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767120.053082:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767120.053089:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066497664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767120.053096:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767120.053099:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767120.053100:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767120.053103:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3951 00000100:00080000:3.0:1775767120.053105:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767120.053107:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767120.053111:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767120.053117:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066497536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767120.053119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066497792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767120.053130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767120.053131:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767120.053132:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767120.053135:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3951 00000100:00080000:3.0:1775767120.053138:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767120.053140:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767120.053143:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767120.053152:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066497920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767120.053158:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767120.053160:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767120.053161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767120.053163:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3951 00000100:00080000:3.0:1775767120.053167:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767120.053169:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767120.053171:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767120.053177:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767120.053181:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066498048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767120.053229:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767120/real 1775767120] req@ffff9d8188742d80 x1862023066497536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767120.053243:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066497536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767120.053251:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767120.053254:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767120.053257:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066497536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767120.053262:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767120.053285:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767120.053292:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767120/real 1775767120] req@ffff9d8188740a80 x1862023066497664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767120.053300:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066497664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767120.053305:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767120.053307:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767120.053309:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767120.053325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767120.053335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767120.053341:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767120/real 1775767120] req@ffff9d8188741500 x1862023066497792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767120.053348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066497792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767120.053353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767120.053355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767120.053356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767120.053360:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767120.053381:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767120.053388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767120/real 1775767120] req@ffff9d8188743b80 x1862023066497920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767120.053395:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066497920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767120.053401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767120.053403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767120.053405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066497920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767120.053409:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767120.053423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767120.053429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767120/real 1775767120] req@ffff9d8188743480 x1862023066498048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767120.053440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066498048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767120.053446:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767120.053447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767120.053449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767120.693077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767121.077004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767121.717086:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767122.100966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767122.741115:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767123.124968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767123.765134:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767124.148993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767124.789067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767125.173019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767125.173067:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767125.173093:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767125.173098:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767125.173107:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3956 00000100:00080000:1.0:1775767125.173118:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767125.173125:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767125.173177:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066498176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767125.173206:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767125.173210:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767125.173212:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767125.173215:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3956 00000100:00080000:1.0:1775767125.173219:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767125.173222:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767125.173231:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767125.173273:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066498304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767125.173288:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767125.173289:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066498176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767125.173297:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767125.173302:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767125.173309:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3956 00000100:00080000:1.0:1775767125.173312:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767125.173316:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767125.173321:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767125.173332:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066498432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767125.173341:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767125.173345:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767125.173347:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767125.173350:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3956 00000100:00080000:1.0:1775767125.173376:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767125.173381:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767125.173386:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767125.173396:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066498560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767125.173406:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767125.173410:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767125.173412:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767125.173417:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3956 00000100:00080000:1.0:1775767125.173422:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767125.173425:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767125.173430:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767125.173444:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066498688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767125.173470:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767125.173542:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767125.173582:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767125.173674:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767125.813060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767126.197144:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767126.837080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767127.221136:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767127.861115:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767128.245099:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767128.245246:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767128.245287:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767128.245305:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767128.245346:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767128.245353:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767128.245362:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767128.245372:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767128.245400:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767125/real 1775767128] req@ffff9d8190553480 x1862023066498304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767128.245429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066498304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767128.245441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767128.245447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767128.245451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767128.245460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767125/real 1775767128] req@ffff9d8190553100 x1862023066498432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767128.245471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066498432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767128.245479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767128.245482:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767128.245485:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767128.245491:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767125/real 1775767128] req@ffff9d8190551500 x1862023066498560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767128.245512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066498560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767128.245521:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767128.245524:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767128.245526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767128.245533:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767125/real 1775767128] req@ffff9d8190551880 x1862023066498688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767128.245544:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066498688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767128.245551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767128.245554:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767128.245558:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498688:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767128.245565:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767125/real 1775767128] req@ffff9d8190550e00 x1862023066498176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767128.245593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066498176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767128.245604:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767128.245607:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767128.245610:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066498176:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775767128.885100:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767129.269057:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767129.909077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767130.292952:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767130.292960:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767130.292963:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767130.292971:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3961 00000100:00080000:1.0:1775767130.292980:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767130.292986:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767130.293017:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066498816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775767130.293017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767130.293046:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767130.293049:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767130.293050:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767130.293053:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3961 00000100:00080000:1.0:1775767130.293056:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767130.293059:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767130.293065:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767130.293073:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066498944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767130.293081:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767130.293083:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767130.293085:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767130.293088:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3961 00000100:00080000:1.0:1775767130.293090:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767130.293092:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767130.293096:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767130.293099:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066498816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767130.293105:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066499072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767130.293112:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767130.293114:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767130.293116:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767130.293118:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3961 00000100:00080000:1.0:1775767130.293121:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767130.293123:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767130.293126:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767130.293135:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066499200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767130.293144:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767130.293146:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767130.293148:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767130.293150:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3961 00000100:00080000:1.0:1775767130.293153:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767130.293155:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767130.293157:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767130.293166:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066499328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767130.293173:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767130.293218:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767130/real 1775767130] req@ffff9d8190550e00 x1862023066498816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767130.293238:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066498816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767130.293249:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767130.293253:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767130.293257:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066498816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767130.293264:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767130.293287:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767130.293300:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767130/real 1775767130] req@ffff9d8190551880 x1862023066498944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767130.293313:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066498944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767130.293321:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767130.293324:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767130.293327:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066498944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767130.293350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767130.293370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767130.293380:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767130/real 1775767130] req@ffff9d8190551500 x1862023066499072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767130.293391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066499072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767130.293400:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767130.293404:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767130.293407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767130.293414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767130.293433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767130.293443:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767130/real 1775767130] req@ffff9d8190553100 x1862023066499200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767130.293473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066499200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767130.293482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767130.293486:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767130.293489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767130.293496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767130.293517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767130.293528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767130/real 1775767130] req@ffff9d8190553480 x1862023066499328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767130.293542:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066499328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767130.293550:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767130.293586:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767130.293590:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767130.933061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767131.317064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767131.957106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767132.340954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767132.981080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767133.365084:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767134.005122:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767134.389021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767135.029046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767135.412973:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767135.412981:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775767135.412985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767135.412985:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767135.412994:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3966 00000100:00080000:3.0:1775767135.413003:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767135.413009:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767135.413040:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066499456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767135.413061:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767135.413065:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767135.413067:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767135.413071:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3966 00000100:00080000:3.0:1775767135.413074:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767135.413078:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767135.413088:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767135.413100:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066499584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767135.413112:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767135.413114:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066499456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767135.413115:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767135.413117:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767135.413121:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3966 00000100:00080000:3.0:1775767135.413124:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767135.413128:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767135.413132:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767135.413141:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066499712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767135.413149:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767135.413152:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767135.413154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767135.413157:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3966 00000100:00080000:3.0:1775767135.413159:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767135.413162:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767135.413166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767135.413169:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767135.413174:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066499840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767135.413182:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767135.413185:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767135.413187:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767135.413190:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3966 00000100:00080000:3.0:1775767135.413192:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767135.413195:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767135.413199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767135.413208:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066499968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767135.413210:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767135/real 1775767135] req@ffff9d8188743100 x1862023066499456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767135.413228:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066499456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767135.413239:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767135.413243:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767135.413247:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066499456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767135.413266:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767135.413287:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767135.413299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767135/real 1775767135] req@ffff9d8188741180 x1862023066499584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767135.413311:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066499584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767135.413320:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767135.413323:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767135.413327:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767135.413333:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767135.413351:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767135.413359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767135/real 1775767135] req@ffff9d8188740e00 x1862023066499712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767135.413367:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066499712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767135.413372:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767135.413374:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767135.413376:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767135.413380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767135.413391:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767135.413397:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767135/real 1775767135] req@ffff9d8188742680 x1862023066499840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767135.413409:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066499840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767135.413414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767135.413416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767135.413418:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767135.413421:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767135.413431:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767135.413436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767135/real 1775767135] req@ffff9d8188741880 x1862023066499968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767135.413443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066499968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767135.413448:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767135.413450:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767135.413451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066499968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767136.053027:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767136.436960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767137.076986:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767137.460975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767138.101170:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767138.484978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767139.125171:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767139.509006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767140.149003:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767140.532935:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767140.532950:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767140.532958:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767140.532962:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767140.532969:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3972 00000100:00080000:3.0:1775767140.532978:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767140.533010:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767140.533055:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066500096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767140.533086:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767140.533089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767140.533091:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767140.533105:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3972 00000100:00080000:3.0:1775767140.533108:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767140.533112:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767140.533121:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767140.533136:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066500224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767140.533148:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767140.533150:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066500096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767140.533157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767140.533160:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767140.533164:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3972 00000100:00080000:3.0:1775767140.533166:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767140.533180:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767140.533184:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767140.533193:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066500352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767140.533200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767140.533202:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767140.533204:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767140.533206:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3972 00000100:00080000:3.0:1775767140.533209:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767140.533211:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767140.533214:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767140.533223:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066500480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767140.533231:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767140.533234:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767140.533236:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767140.533239:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767140.533244:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3972 00000100:00080000:3.0:1775767140.533250:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767140.533255:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767140.533262:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767140.533276:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066500608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767140.533303:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767140/real 1775767140] req@ffff9d8188743800 x1862023066500096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767140.533320:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066500096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767140.533329:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767140.533332:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767140.533336:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066500096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767140.533392:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767140.533415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767140.533425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767140/real 1775767140] req@ffff9d8188741f80 x1862023066500224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767140.533443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066500224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767140.533463:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767140.533466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767140.533468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767140.533484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767140.533500:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767140.533507:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767140/real 1775767140] req@ffff9d8188742300 x1862023066500352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767140.533515:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066500352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767140.533521:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767140.533523:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767140.533526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767140.533530:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767140.533540:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767140.533546:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767140/real 1775767140] req@ffff9d8188740000 x1862023066500480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767140.533553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066500480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767140.533559:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767140.533561:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767140.533563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767140.533566:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767140.533576:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767140.533581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767140/real 1775767140] req@ffff9d8188740700 x1862023066500608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767140.533588:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066500608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767140.533594:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767140.533595:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767140.533597:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767141.173150:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767141.556964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767142.197064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767142.580998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767143.221089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767143.605002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767144.245076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767144.629128:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767145.269027:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767145.652971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767145.653040:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767145.653047:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767145.653050:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767145.653058:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3977 00000100:00080000:0.0:1775767145.653067:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767145.653072:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767145.653110:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066500736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767145.653137:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767145.653140:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767145.653142:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767145.653145:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3977 00000100:00080000:0.0:1775767145.653147:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767145.653150:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767145.653158:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767145.653170:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066500864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767145.653174:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066500736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767145.653193:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767145.653198:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767145.653200:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767145.653206:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3977 00000100:00080000:0.0:1775767145.653211:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767145.653216:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767145.653222:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767145.653237:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767145.653240:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066500992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767145.653271:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767145.653274:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767145.653276:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767145.653281:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3977 00000100:00080000:0.0:1775767145.653294:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767145.653300:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767145.653300:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767145/real 1775767145] req@ffff9d81901ad180 x1862023066500736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767145.653305:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767145.653318:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066501120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767145.653341:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767145.653344:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767145.653345:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767145.653349:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3977 00000100:00080000:0.0:1775767145.653352:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767145.653358:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767145.653359:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066500736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767145.653362:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767145.653374:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066501248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767145.653375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767145.653385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767145.653391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066500736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767145.653455:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767145.653479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767145.653493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767145/real 1775767145] req@ffff9d81901ae680 x1862023066500864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767145.653506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066500864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767145.653512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767145.653515:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767145.653517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767145.653521:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767145.653536:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767145.653542:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767145/real 1775767145] req@ffff9d81901aed80 x1862023066500992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767145.653550:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066500992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767145.653555:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767145.653557:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767145.653559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066500992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767145.653563:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767145.653573:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767145.653578:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767145/real 1775767145] req@ffff9d81901afb80 x1862023066501120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767145.653585:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066501120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767145.653591:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767145.653593:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767145.653595:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767145.653599:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767145.653608:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767145.653614:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767145/real 1775767145] req@ffff9d81901af100 x1862023066501248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767145.653620:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066501248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767145.653626:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767145.653627:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767145.653629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767146.293100:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767146.677047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767147.317139:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767147.700936:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767148.341060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767148.725055:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767149.365005:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767149.748972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767150.389126:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767150.772987:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767150.773004:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767150.773011:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767150.773036:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3982 00000100:00080000:1.0:1775767150.773054:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767150.773063:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767150.773073:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775767150.773127:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066501376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767150.773159:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767150.773163:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767150.773164:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767150.773167:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3982 00000100:00080000:1.0:1775767150.773170:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767150.773173:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767150.773184:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767150.773196:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066501504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767150.773204:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767150.773207:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767150.773208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767150.773211:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3982 00000100:00080000:1.0:1775767150.773213:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767150.773215:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767150.773218:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767150.773225:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066501632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767150.773231:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767150.773234:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767150.773236:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767150.773239:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3982 00000100:00080000:1.0:1775767150.773241:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767150.773243:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767150.773247:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767150.773253:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066501760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767150.773261:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066501376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767150.773263:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767150.773271:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767150.773273:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767150.773278:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3982 00000100:00080000:1.0:1775767150.773282:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767150.773287:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767150.773292:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767150.773306:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066501888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767150.773378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767150.773437:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767150/real 1775767150] req@ffff9d8190553480 x1862023066501376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767150.773455:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066501376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767150.773471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767150.773475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767150.773479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066501376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767150.773485:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767150.773504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767150.773514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767150/real 1775767150] req@ffff9d8190553100 x1862023066501504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767150.773522:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066501504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767150.773529:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767150.773531:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767150.773533:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767150.773537:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767150.773547:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767150.773552:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767150/real 1775767150] req@ffff9d8190551500 x1862023066501632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767150.773560:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066501632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767150.773565:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767150.773567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767150.773569:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767150.773616:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767150.773637:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767150.773648:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767150/real 1775767150] req@ffff9d8190551880 x1862023066501760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767150.773662:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066501760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767150.773673:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767150.773677:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767150.773681:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767150.773687:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767150.773715:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767150.773721:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767150/real 1775767150] req@ffff9d8190550e00 x1862023066501888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767150.773728:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066501888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767150.773734:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767150.773736:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767150.773738:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066501888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767151.413014:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767151.796988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767152.437097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767152.821089:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767153.461049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767153.845009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767154.485067:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767154.868991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767155.509044:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767155.893029:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767155.893033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767155.893044:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767155.893049:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767155.893061:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3987 00000100:00080000:1.0:1775767155.893073:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767155.893080:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767155.893135:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066502016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767155.893170:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767155.893173:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767155.893174:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767155.893177:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3987 00000100:00080000:1.0:1775767155.893180:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767155.893182:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767155.893204:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767155.893215:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066502144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767155.893228:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767155.893232:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767155.893235:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767155.893237:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066502016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767155.893241:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3987 00000100:00080000:1.0:1775767155.893246:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767155.893248:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767155.893251:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767155.893260:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066502272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767155.893266:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767155.893269:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767155.893271:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767155.893273:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3987 00000100:00080000:1.0:1775767155.893275:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767155.893278:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767155.893281:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767155.893307:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066502400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767155.893317:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767155.893321:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767155.893322:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767155.893336:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3987 00000100:00080000:1.0:1775767155.893341:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767155.893345:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775767155.893350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775767155.893352:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767155.893365:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066502528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767155.893415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767155/real 1775767155] req@ffff9d8190550e00 x1862023066502016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767155.893432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066502016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767155.893445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767155.893449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767155.893452:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066502016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767155.893474:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767155.893490:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767155.893499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767155/real 1775767155] req@ffff9d8190551880 x1862023066502144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767155.893507:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066502144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767155.893512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767155.893514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767155.893516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767155.893520:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767155.893531:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767155.893537:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767155/real 1775767155] req@ffff9d8190551500 x1862023066502272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767155.893544:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066502272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767155.893550:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767155.893552:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767155.893554:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767155.893558:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767155.893569:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767155.893574:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767155/real 1775767155] req@ffff9d8190553100 x1862023066502400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767155.893581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066502400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767155.893587:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767155.893589:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767155.893591:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767155.893594:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767155.893603:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767155.893608:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767155/real 1775767155] req@ffff9d8190553480 x1862023066502528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767155.893616:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066502528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767155.893621:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767155.893623:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767155.893625:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767156.533059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767156.916990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767157.557031:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767157.940990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767158.581112:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767158.964986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767159.605081:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767159.988962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767160.629104:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767161.013036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767161.013084:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767161.013092:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767161.013097:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767161.013107:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3992 00000100:00080000:3.0:1775767161.013116:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767161.013123:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767161.013175:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066502656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767161.013205:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767161.013208:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767161.013210:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767161.013213:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3992 00000100:00080000:3.0:1775767161.013215:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767161.013219:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767161.013227:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767161.013288:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066502784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767161.013294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066502656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767161.013321:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767161.013323:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767161.013325:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767161.013329:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3992 00000100:00080000:3.0:1775767161.013333:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767161.013337:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767161.013343:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767161.013354:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066502912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767161.013363:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767161.013365:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767161.013367:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767161.013370:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3992 00000100:00080000:3.0:1775767161.013386:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767161.013389:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767161.013394:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767161.013402:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767161.013421:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066503040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767161.013430:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767161.013433:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767161.013436:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767161.013441:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3992 00000100:00080000:3.0:1775767161.013443:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767161.013448:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767161.013453:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767161.013467:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066503168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767161.013478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767161/real 1775767161] req@ffff9d8188742a00 x1862023066502656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767161.013499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066502656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767161.013508:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767161.013513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767161.013516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066502656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767161.013540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767161.013560:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767161.013568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767161/real 1775767161] req@ffff9d8188741c00 x1862023066502784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767161.013577:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066502784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767161.013583:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767161.013586:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767161.013588:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767161.013593:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767161.013604:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767161.013610:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767161/real 1775767161] req@ffff9d8188740380 x1862023066502912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767161.013618:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066502912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767161.013623:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767161.013625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767161.013627:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066502912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767161.013631:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767161.013642:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767161.013648:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767161/real 1775767161] req@ffff9d8188740700 x1862023066503040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767161.013656:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066503040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767161.013662:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767161.013664:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767161.013692:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767161.013697:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767161.013712:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767161.013718:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767161/real 1775767161] req@ffff9d8188740000 x1862023066503168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767161.013727:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066503168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767161.013749:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767161.013752:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767161.013754:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767161.653020:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767162.036959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767162.677073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767163.061012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767163.701053:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767164.085053:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767164.725309:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767165.109017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767165.749061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767166.132976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767166.133032:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767166.133037:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767166.133040:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767166.133048:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 3997 00000100:00080000:1.0:1775767166.133057:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767166.133062:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767166.133098:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066503296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767166.133117:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767166.133120:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767166.133121:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767166.133124:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3997 00000100:00080000:1.0:1775767166.133127:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767166.133130:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767166.133139:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767166.133147:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066503296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767166.133154:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066503424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767166.133166:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767166.133172:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767166.133173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767166.133178:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3997 00000100:00080000:1.0:1775767166.133199:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767166.133203:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767166.133209:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767166.133224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767166.133225:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066503552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767166.133238:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767166.133241:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767166.133243:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767166.133257:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3997 00000100:00080000:1.0:1775767166.133264:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767166.133270:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767166/real 1775767166] req@ffff9d8190553480 x1862023066503296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767166.133271:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767166.133280:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767166.133292:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066503296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767166.133297:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066503680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767166.133304:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767166.133313:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:1.0:1775767166.133318:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767166.133319:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066503296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767166.133327:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767166.133330:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767166.133333:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 3997 00000100:00080000:1.0:1775767166.133344:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767166.133346:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503424:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775767166.133350:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767166.133356:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767166.133368:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066503808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767166.133373:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767166.133386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767166/real 1775767166] req@ffff9d8190553100 x1862023066503424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767166.133398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066503424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767166.133405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767166.133407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767166.133410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767166.133413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767166.133424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767166.133430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767166/real 1775767166] req@ffff9d8190551500 x1862023066503552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767166.133437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066503552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767166.133442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767166.133444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767166.133446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767166.133449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767166.133460:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767166.133465:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767166/real 1775767166] req@ffff9d8190551880 x1862023066503680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767166.133472:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066503680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767166.133478:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767166.133480:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767166.133482:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767166.133490:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767166.133500:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767166.133514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767166/real 1775767166] req@ffff9d8190550e00 x1862023066503808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767166.133521:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066503808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767166.133527:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767166.133529:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767166.133531:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066503808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767166.773009:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767167.156940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767167.797143:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767168.180982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767168.821053:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767169.205014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767169.845056:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767170.228981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767170.869040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767171.253028:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767171.253033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767171.253036:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767171.253049:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767171.253060:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4002 00000100:00080000:2.0:1775767171.253072:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767171.253078:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767171.253114:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066503936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767171.253156:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767171.253160:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767171.253161:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767171.253165:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4002 00000100:00080000:2.0:1775767171.253188:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767171.253192:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767171.253200:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767171.253208:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066503936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767171.253215:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066504064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767171.253224:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767171.253226:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767171.253228:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767171.253233:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4002 00000100:00080000:2.0:1775767171.253235:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767171.253238:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767171.253241:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767171.253251:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066504192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767171.253258:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767171.253262:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767171.253263:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767171.253266:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4002 00000100:00080000:2.0:1775767171.253279:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767171.253282:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767171.253286:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767171.253291:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767171.253294:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066504320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767171.253302:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767171.253305:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767171.253306:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767171.253310:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4002 00000100:00080000:2.0:1775767171.253317:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767171.253320:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767171.253324:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767171.253334:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066504448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767171.253352:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767171/real 1775767171] req@ffff9d81984a0a80 x1862023066503936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767171.253371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066503936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767171.253383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767171.253389:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767171.253393:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066503936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767171.253422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767171.253448:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767171.253458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767171/real 1775767171] req@ffff9d81984a1500 x1862023066504064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767171.253468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066504064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767171.253474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767171.253477:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767171.253479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767171.253484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767171.253497:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767171.253512:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767171/real 1775767171] req@ffff9d81984a3100 x1862023066504192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767171.253525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066504192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767171.253530:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767171.253532:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767171.253534:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767171.253538:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767171.253551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767171.253557:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767171/real 1775767171] req@ffff9d81984a3800 x1862023066504320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767171.253576:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066504320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767171.253582:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767171.253584:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767171.253585:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767171.253589:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767171.253599:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767171.253605:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767171/real 1775767171] req@ffff9d81984a1880 x1862023066504448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767171.253612:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066504448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767171.253623:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767171.253626:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767171.253638:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767171.893064:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767172.277000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767172.917080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767173.301001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767173.941042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767174.324994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767174.965053:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767175.349001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767175.989142:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767176.372969:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767176.372975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767176.372996:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767176.373002:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767176.373015:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4007 00000100:00080000:2.0:1775767176.373027:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767176.373036:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767176.373085:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066504576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767176.373119:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767176.373123:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767176.373125:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767176.373129:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4007 00000100:00080000:2.0:1775767176.373132:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767176.373135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767176.373142:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767176.373154:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066504704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767176.373162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767176.373166:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767176.373168:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767176.373171:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066504576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767176.373174:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4007 00000100:00080000:2.0:1775767176.373180:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767176.373182:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767176.373186:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767176.373195:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066504832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767176.373202:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767176.373205:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767176.373206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767176.373209:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4007 00000100:00080000:2.0:1775767176.373227:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767176.373229:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767176.373232:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767176.373237:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767176.373242:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066504960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767176.373253:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767176.373257:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767176.373259:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767176.373263:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4007 00000100:00080000:2.0:1775767176.373267:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767176.373272:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767176.373277:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767176.373284:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767176/real 1775767176] req@ffff9d81984a1180 x1862023066504576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767176.373292:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066505088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767176.373304:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066504576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767176.373314:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767176.373317:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767176.373322:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066504576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767176.373340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767176.373355:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767176.373364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767176/real 1775767176] req@ffff9d81984a0000 x1862023066504704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767176.373376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066504704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767176.373381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767176.373383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767176.373392:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767176.373396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767176.373407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767176.373413:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767176/real 1775767176] req@ffff9d81984a3b80 x1862023066504832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767176.373421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066504832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767176.373426:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767176.373429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767176.373431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767176.373434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767176.373443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767176.373448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767176/real 1775767176] req@ffff9d81984a1f80 x1862023066504960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767176.373455:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066504960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767176.373466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767176.373468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767176.373471:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066504960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767176.373477:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767176.373494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767176.373504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767176/real 1775767176] req@ffff9d81984a0e00 x1862023066505088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767176.373516:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066505088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767176.373536:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767176.373539:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767176.373541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767177.013060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767177.397047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767178.037143:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767178.420980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767179.061063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767179.444946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767180.085124:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767180.468997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767181.109087:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767181.493007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767181.493009:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767181.493046:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767181.493051:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767181.493063:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4013 00000100:00080000:2.0:1775767181.493075:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767181.493084:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767181.493134:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066505216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767181.493162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767181.493165:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767181.493167:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767181.493170:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4013 00000100:00080000:2.0:1775767181.493172:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767181.493175:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767181.493191:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767181.493199:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066505216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767181.493204:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066505344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767181.493213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767181.493216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767181.493218:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767181.493221:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4013 00000100:00080000:2.0:1775767181.493225:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767181.493228:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767181.493232:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767181.493242:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066505472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767181.493249:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767181.493252:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767181.493253:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767181.493255:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4013 00000100:00080000:2.0:1775767181.493271:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767181.493274:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767181.493277:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767181.493286:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066505600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767181.493292:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775767181.493298:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767181.493301:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767181.493303:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767181.493306:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4013 00000100:00080000:2.0:1775767181.493309:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767181.493311:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767181.493314:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767181.493321:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066505728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767181.493365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767181/real 1775767181] req@ffff9d81984a2680 x1862023066505216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767181.493397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066505216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767181.493407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767181.493410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767181.493414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066505216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767181.493446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767181.493467:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767181.493477:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767181/real 1775767181] req@ffff9d81984a3480 x1862023066505344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767181.493486:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066505344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767181.493492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767181.493494:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767181.493496:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767181.493500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767181.493511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767181.493517:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767181/real 1775767181] req@ffff9d81984a2d80 x1862023066505472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767181.493525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066505472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767181.493534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767181.493537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767181.493539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767181.493542:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767181.493554:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767181.493559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767181/real 1775767181] req@ffff9d81984a0700 x1862023066505600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767181.493566:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066505600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767181.493572:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767181.493574:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767181.493576:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767181.493579:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767181.493589:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767181.493595:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767181/real 1775767181] req@ffff9d81984a2a00 x1862023066505728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767181.493602:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066505728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767181.493608:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767181.493610:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767181.493612:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767182.133137:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767182.516996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767183.157148:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767183.540983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767184.181141:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767184.565035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767185.205189:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767185.589062:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767186.229169:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767186.612982:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767186.612994:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767186.612999:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767186.613009:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4018 00000100:00080000:2.0:1775767186.613021:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767186.613029:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767186.613031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:2.0:1775767186.613075:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066505856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767186.613105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767186.613108:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767186.613110:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767186.613112:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4018 00000100:00080000:2.0:1775767186.613115:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767186.613118:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767186.613128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767186.613142:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066505856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767186.613150:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066505984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767186.613167:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767186.613171:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767186.613174:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767186.613180:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4018 00000100:00080000:2.0:1775767186.613198:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767186.613204:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767186.613208:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767186.613217:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767186.613233:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066506112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767186.613244:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767186.613247:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767186.613249:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767186.613256:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4018 00000100:00080000:2.0:1775767186.613261:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767186.613262:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767186/real 1775767186] req@ffff9d81984a2300 x1862023066505856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767186.613273:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767186.613286:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767186.613291:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066505856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767186.613300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767186.613305:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:2.0:1775767186.613307:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066506240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767186.613311:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066505856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767186.613331:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767186.613335:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767186.613337:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767186.613343:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4018 00000100:00100000:0.0:1775767186.613345:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505984:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775767186.613350:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767186.613357:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767186.613362:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767186.613378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767186.613381:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066506368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767186.613399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767186/real 1775767186] req@ffff9d81984a2a00 x1862023066505984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767186.613414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066505984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767186.613422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767186.613425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767186.613427:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066505984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767186.613432:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767186.613449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767186.613456:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767186/real 1775767186] req@ffff9d81984a0700 x1862023066506112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767186.613464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066506112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767186.613468:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767186.613470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767186.613472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767186.613476:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767186.613487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767186.613492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767186/real 1775767186] req@ffff9d81984a2d80 x1862023066506240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767186.613499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066506240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767186.613507:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767186.613508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767186.613510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767186.613521:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767186.613532:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767186.613538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767186/real 1775767186] req@ffff9d81984a3480 x1862023066506368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767186.613545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066506368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767186.613550:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767186.613552:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767186.613554:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767187.253082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767187.637018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767188.277225:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767188.660993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767189.301118:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767189.685018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767190.325112:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767190.708995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767191.349038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767191.732985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767191.733006:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767191.733015:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767191.733021:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767191.733038:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4023 00000100:00080000:2.0:1775767191.733050:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767191.733058:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767191.733103:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066506496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767191.733135:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767191.733138:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767191.733140:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767191.733143:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4023 00000100:00080000:2.0:1775767191.733145:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767191.733148:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767191.733157:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767191.733169:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066506624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767191.733176:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066506496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767191.733189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767191.733194:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767191.733195:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767191.733199:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4023 00000100:00080000:2.0:1775767191.733202:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767191.733205:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767191.733210:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767191.733222:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066506752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767191.733233:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767191.733236:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767191.733237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767191.733240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4023 00000100:00080000:2.0:1775767191.733246:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767191.733248:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767191.733252:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767191.733260:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066506880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767191.733266:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767191.733269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767191.733271:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767191.733274:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4023 00000100:00080000:2.0:1775767191.733276:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767191.733278:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767191.733282:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767191.733296:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066507008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767191.733324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767191.733354:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767191.733373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767191.733397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767192.373085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767192.756974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767193.397060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767193.780961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767194.421058:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767194.804977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767194.805124:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767194.805158:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767194.805178:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767194.805223:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767194.805230:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767194.805235:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767194.805243:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767194.805302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767191/real 1775767194] req@ffff9d81984a0e00 x1862023066506624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767194.805328:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066506624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767194.805340:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767194.805346:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767194.805350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767194.805358:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767191/real 1775767194] req@ffff9d81984a1f80 x1862023066506752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767194.805371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066506752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767194.805376:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767194.805379:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767194.805381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767194.805388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767191/real 1775767194] req@ffff9d81984a3b80 x1862023066506880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767194.805412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066506880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767194.805422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767194.805425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767194.805428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066506880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767194.805440:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767191/real 1775767194] req@ffff9d81984a0000 x1862023066507008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767194.805453:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066507008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767194.805462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767194.805465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767194.805469:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507008:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767194.805474:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767191/real 1775767194] req@ffff9d81984a2680 x1862023066506496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767194.805485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066506496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767194.805502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767194.805506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767194.805509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066506496:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775767195.445166:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767195.828974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767196.469060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767196.852968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767196.852990:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767196.852998:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767196.853003:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767196.853013:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4028 00000100:00080000:2.0:1775767196.853026:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767196.853034:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767196.853079:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066507136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767196.853110:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767196.853115:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767196.853117:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767196.853122:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4028 00000100:00080000:2.0:1775767196.853128:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767196.853133:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767196.853147:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066507136:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775767196.853148:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767196.853179:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066507264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767196.853194:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767196.853199:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767196.853201:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767196.853206:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4028 00000100:00080000:2.0:1775767196.853212:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767196.853218:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767196.853223:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767196.853247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767196.853269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066507392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767196.853279:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767196/real 1775767196] req@ffff9d81984a1180 x1862023066507136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767196.853284:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767196.853303:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767196.853306:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767196.853314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066507136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767196.853315:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4028 00000100:00080000:2.0:1775767196.853325:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767196.853335:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775767196.853338:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775767196.853345:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775767196.853346:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767196.853358:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066507136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767196.853370:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066507520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767196.853383:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767196.853391:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507264:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775767196.853402:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767196.853405:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767196.853413:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4028 00000100:00080000:2.0:1775767196.853423:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767196.853428:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775767196.853438:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767196.853446:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767196.853452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767196/real 1775767196] req@ffff9d81984a1880 x1862023066507264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767196.853464:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066507648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767196.853475:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066507264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767196.853500:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767196.853505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767196.853510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767196.853520:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767196.853552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767196.853564:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767196/real 1775767196] req@ffff9d81984a3800 x1862023066507392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767196.853577:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066507392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767196.853585:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767196.853589:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767196.853592:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767196.853600:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767196.853624:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767196.853635:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767196/real 1775767196] req@ffff9d81984a3100 x1862023066507520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767196.853647:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066507520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767196.853658:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767196.853662:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767196.853665:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767196.853684:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767196.853719:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767196.853731:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767196/real 1775767196] req@ffff9d81984a1500 x1862023066507648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767196.853743:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066507648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767196.853752:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767196.853756:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767196.853759:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767197.493119:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767197.877017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767198.517072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767198.900988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767199.541112:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767199.925054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767200.565054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767200.949010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767201.589144:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767201.972952:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767201.972961:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767201.972965:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767201.972975:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4033 00000100:00080000:2.0:1775767201.972985:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767201.972992:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767201.972993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:2.0:1775767201.973025:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066507776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767201.973044:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767201.973047:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767201.973049:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767201.973053:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4033 00000100:00080000:2.0:1775767201.973056:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767201.973059:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767201.973066:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767201.973074:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066507776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767201.973080:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066507904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767201.973089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767201.973092:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767201.973093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767201.973096:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4033 00000100:00080000:2.0:1775767201.973098:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767201.973100:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767201.973103:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767201.973111:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066508032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767201.973117:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767201.973120:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767201.973121:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767201.973123:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4033 00000100:00080000:2.0:1775767201.973143:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767201.973146:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767201.973150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767201.973155:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767201.973161:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066508160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767201.973168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767201.973170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767201.973171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767201.973174:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4033 00000100:00080000:2.0:1775767201.973177:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767201.973181:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767201.973185:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767201.973193:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066508288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767201.973247:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767201/real 1775767201] req@ffff9d81984a0a80 x1862023066507776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767201.973269:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066507776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767201.973281:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767201.973286:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767201.973291:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066507776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767201.973321:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767201.973343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767201.973353:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767201/real 1775767201] req@ffff9d81984a1500 x1862023066507904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767201.973362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066507904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767201.973369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767201.973371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767201.973373:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066507904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767201.973380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767201.973392:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767201.973398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767201/real 1775767201] req@ffff9d81984a3100 x1862023066508032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767201.973406:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066508032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767201.973412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767201.973414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767201.973415:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767201.973419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767201.973431:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767201.973437:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767201/real 1775767201] req@ffff9d81984a3800 x1862023066508160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767201.973444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066508160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767201.973450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767201.973452:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767201.973454:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767201.973457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767201.973468:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767201.973473:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767201/real 1775767201] req@ffff9d81984a1880 x1862023066508288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767201.973481:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066508288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767201.973488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767201.973489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767201.973491:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767202.613083:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767202.997030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767203.637065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767204.020975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767204.661104:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767205.044998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767205.685188:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767206.069016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767206.709053:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767207.093006:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767207.093015:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767207.093037:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767207.093041:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767207.093049:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4038 00000100:00080000:1.0:1775767207.093062:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767207.093071:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767207.093113:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066508416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767207.093141:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767207.093144:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767207.093146:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767207.093149:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4038 00000100:00080000:1.0:1775767207.093152:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767207.093156:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767207.093166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767207.093180:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066508544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767207.093182:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066508416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767207.093219:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767207.093222:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767207.093223:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767207.093227:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4038 00000100:00080000:1.0:1775767207.093229:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767207.093232:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767207.093243:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767207.093253:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066508672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767207.093260:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767207.093262:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767207.093264:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767207.093267:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4038 00000100:00080000:1.0:1775767207.093285:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767207.093287:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767207.093291:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767207.093299:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767207.093300:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066508800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767207.093311:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767207.093314:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767207.093316:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767207.093319:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4038 00000100:00080000:1.0:1775767207.093325:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767207.093328:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767207.093332:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767207.093341:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066508928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767207.093368:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767207/real 1775767207] req@ffff9d8190550e00 x1862023066508416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767207.093388:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066508416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767207.093397:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767207.093401:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767207.093405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066508416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767207.093428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767207.093446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767207.093455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767207/real 1775767207] req@ffff9d8190551880 x1862023066508544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767207.093464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066508544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767207.093470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767207.093472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767207.093475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767207.093479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767207.093504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767207.093515:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767207/real 1775767207] req@ffff9d8190551500 x1862023066508672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767207.093528:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066508672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767207.093537:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767207.093541:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767207.093544:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767207.093551:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767207.093568:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767207.093576:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767207/real 1775767207] req@ffff9d8190553100 x1862023066508800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767207.093584:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066508800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767207.093591:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767207.093593:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767207.093596:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767207.093599:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767207.093609:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767207.093615:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767207/real 1775767207] req@ffff9d8190553480 x1862023066508928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767207.093622:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066508928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767207.093628:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767207.093630:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767207.093632:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066508928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767207.733070:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767208.117043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767208.757140:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767209.140982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767209.781033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767210.165047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767210.805113:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767211.189004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767211.829029:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767212.213004:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767212.213015:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767212.213020:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767212.213031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767212.213032:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4043 00000100:00080000:1.0:1775767212.213047:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767212.213057:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767212.213111:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066509056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767212.213140:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767212.213143:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767212.213145:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767212.213149:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4043 00000100:00080000:1.0:1775767212.213154:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767212.213160:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767212.213173:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767212.213177:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066509056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767212.213198:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066509184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767212.213219:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767212.213222:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767212.213224:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767212.213228:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4043 00000100:00080000:1.0:1775767212.213241:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767212.213246:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767212.213251:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767212.213256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767212.213271:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066509312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767212.213280:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767212.213283:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767212.213286:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767212.213291:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4043 00000100:00080000:1.0:1775767212.213295:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767212.213299:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767212.213304:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767212/real 1775767212] req@ffff9d8190553480 x1862023066509056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767212.213307:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767212.213324:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066509440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767212.213355:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767212.213357:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066509056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767212.213369:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767212.213376:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767212.213379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767212.213386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:1.0:1775767212.213388:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4043 00000100:00100000:0.0:1775767212.213395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066509056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767212.213398:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767212.213407:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767212.213412:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767212.213421:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775767212.213429:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066509568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767212.213446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767212.213458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767212/real 1775767212] req@ffff9d8190553100 x1862023066509184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767212.213467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066509184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767212.213473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767212.213476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767212.213478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767212.213482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767212.213495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767212.213504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767212/real 1775767212] req@ffff9d8190551500 x1862023066509312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767212.213526:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066509312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767212.213540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767212.213544:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767212.213547:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767212.213554:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767212.213575:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767212.213587:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767212/real 1775767212] req@ffff9d8190551880 x1862023066509440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767212.213600:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066509440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767212.213607:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767212.213609:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767212.213611:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767212.213623:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767212.213638:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767212.213645:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767212/real 1775767212] req@ffff9d8190550e00 x1862023066509568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767212.213653:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066509568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767212.213658:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767212.213660:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767212.213663:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767212.853064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767213.237014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767213.877053:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767214.260961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767214.901018:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767215.284996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767215.925050:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767216.308979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767216.949083:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767217.333004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767217.333037:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767217.333043:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767217.333047:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767217.333056:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4048 00000100:00080000:3.0:1775767217.333065:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767217.333072:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767217.333110:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066509696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767217.333138:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767217.333142:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767217.333144:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767217.333148:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4048 00000100:00080000:3.0:1775767217.333153:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767217.333157:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767217.333168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767217.333183:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066509696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767217.333184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066509824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767217.333200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767217.333203:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767217.333206:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767217.333210:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4048 00000100:00080000:3.0:1775767217.333215:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767217.333219:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767217.333224:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767217.333238:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066509952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767217.333241:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767217.333253:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767217.333257:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767217.333259:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767217.333264:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4048 00000100:00080000:3.0:1775767217.333268:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767217.333272:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767217.333279:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767217.333284:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767217/real 1775767217] req@ffff9d8188742300 x1862023066509696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767217.333292:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066510080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767217.333302:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066509696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767217.333306:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767217.333315:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767217.333318:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767217.333319:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767217.333328:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775767217.333328:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4048 00000100:00100000:0.0:1775767217.333334:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066509696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767217.333335:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767217.333343:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767217.333348:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767217.333357:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767217.333362:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066510208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767217.333378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767217.333388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767217/real 1775767217] req@ffff9d8188741f80 x1862023066509824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767217.333398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066509824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767217.333404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767217.333407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767217.333409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767217.333413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767217.333423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767217.333438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767217/real 1775767217] req@ffff9d8188743800 x1862023066509952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767217.333449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066509952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767217.333455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767217.333457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767217.333459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066509952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767217.333463:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767217.333475:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767217.333480:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767217/real 1775767217] req@ffff9d8188741880 x1862023066510080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767217.333488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066510080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767217.333494:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767217.333496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767217.333497:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767217.333506:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767217.333516:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767217.333521:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767217/real 1775767217] req@ffff9d8188742680 x1862023066510208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767217.333533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066510208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767217.333538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767217.333540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767217.333542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767217.973009:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767218.356981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767218.997181:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767219.380955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767220.021012:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767220.404955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767221.045109:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767221.428997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767222.069103:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767222.453019:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767222.453029:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767222.453034:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767222.453045:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4053 00000100:00080000:3.0:1775767222.453056:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1775767222.453058:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767222.453065:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767222.453105:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066510336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767222.453134:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767222.453138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767222.453140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767222.453144:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4053 00000100:00080000:3.0:1775767222.453148:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767222.453151:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767222.453162:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767222.453177:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066510464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767222.453188:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066510336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767222.453195:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767222.453201:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767222.453202:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767222.453209:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4053 00000100:00080000:3.0:1775767222.453213:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767222.453217:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767222.453223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767222.453236:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066510592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767222.453245:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767222.453251:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767222.453255:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767222.453258:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767222.453263:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4053 00000100:00080000:3.0:1775767222.453270:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767222.453274:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767222.453280:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767222.453294:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066510720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767222.453297:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767222/real 1775767222] req@ffff9d8188740e00 x1862023066510336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767222.453318:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767222.453322:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767222.453324:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767222.453325:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066510336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767222.453333:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4053 00000100:00080000:0.0:1775767222.453339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767222.453340:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767222.453347:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775767222.453348:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767222.453354:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066510336:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775767222.453358:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767222.453372:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066510848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767222.453374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767222.453398:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767222.453408:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767222/real 1775767222] req@ffff9d8188741180 x1862023066510464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767222.453416:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066510464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767222.453427:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767222.453429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767222.453431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767222.453435:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767222.453445:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767222.453450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767222/real 1775767222] req@ffff9d8188743100 x1862023066510592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767222.453457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066510592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767222.453462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767222.453464:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767222.453466:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767222.453469:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767222.453479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767222.453484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767222/real 1775767222] req@ffff9d8188743480 x1862023066510720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767222.453491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066510720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767222.453497:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767222.453499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767222.453502:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767222.453510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767222.453520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767222.453525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767222/real 1775767222] req@ffff9d8188743b80 x1862023066510848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767222.453532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066510848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767222.453537:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767222.453539:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767222.453541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066510848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767223.093101:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767223.476979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767224.117018:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767224.501004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767225.141060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767225.524959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767226.165045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767226.549018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767227.189096:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767227.572953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767227.572996:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767227.573003:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767227.573007:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767227.573016:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4059 00000100:00080000:1.0:1775767227.573027:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767227.573034:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767227.573083:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066510976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767227.573113:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767227.573117:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767227.573119:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767227.573123:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4059 00000100:00080000:1.0:1775767227.573127:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767227.573131:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767227.573141:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767227.573154:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066511104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767227.573165:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767227.573171:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767227.573173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767227.573178:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4059 00000100:00080000:1.0:1775767227.573181:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767227.573184:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767227.573189:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767227.573201:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066511232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767227.573213:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767227.573219:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066510976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767227.573219:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767227.573221:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767227.573238:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4059 00000100:00080000:1.0:1775767227.573243:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767227.573246:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767227.573253:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767227.573270:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066511360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767227.573281:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767227.573284:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767227.573286:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767227.573289:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4059 00000100:00080000:1.0:1775767227.573303:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767227.573307:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767227.573315:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767227.573321:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767227.573337:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066511488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767227.573398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767227/real 1775767227] req@ffff9d8190550e00 x1862023066510976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767227.573418:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066510976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767227.573428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767227.573432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767227.573435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066510976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767227.573441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767227.573459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767227.573468:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767227/real 1775767227] req@ffff9d8190551880 x1862023066511104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767227.573476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066511104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767227.573482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767227.573485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767227.573487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767227.573508:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767227.573519:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767227.573525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767227/real 1775767227] req@ffff9d8190551500 x1862023066511232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767227.573533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066511232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767227.573538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767227.573540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767227.573542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767227.573546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767227.573572:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767227.573605:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767227/real 1775767227] req@ffff9d8190553100 x1862023066511360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767227.573619:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066511360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767227.573626:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767227.573628:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767227.573631:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767227.573635:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767227.573649:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767227.573655:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767227/real 1775767227] req@ffff9d8190553480 x1862023066511488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767227.573663:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066511488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767227.573668:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767227.573670:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767227.573689:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767228.213050:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767228.596990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767229.237078:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767229.620973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767230.261086:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767230.645013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767231.285065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767231.668969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767232.309014:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767232.692961:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767232.692968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767232.692969:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767232.692973:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767232.692982:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4064 00000100:00080000:1.0:1775767232.692991:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767232.692997:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767232.693032:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066511616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767232.693056:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767232.693059:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767232.693060:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767232.693063:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4064 00000100:00080000:1.0:1775767232.693065:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767232.693067:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767232.693074:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767232.693087:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066511744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767232.693099:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767232.693104:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066511616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767232.693105:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767232.693110:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767232.693117:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4064 00000100:00080000:1.0:1775767232.693122:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767232.693126:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767232.693132:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767232.693146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066511872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767232.693155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767232.693159:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767232.693161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767232.693165:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4064 00000800:00000100:0.0:1775767232.693170:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767232.693171:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767232.693176:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767232.693181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767232.693193:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066512000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767232.693200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767232.693203:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767232.693204:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767232.693206:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4064 00000100:00080000:1.0:1775767232.693210:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767232.693214:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767232.693217:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767232/real 1775767232] req@ffff9d8190553480 x1862023066511616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767232.693222:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767232.693241:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066511616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767232.693243:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066512128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767232.693266:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767232.693271:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767232.693278:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066511616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767232.693302:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767232.693325:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767232.693333:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767232/real 1775767232] req@ffff9d8190553100 x1862023066511744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767232.693342:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066511744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767232.693347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767232.693350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767232.693352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767232.693355:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767232.693365:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767232.693370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767232/real 1775767232] req@ffff9d8190551500 x1862023066511872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767232.693378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066511872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767232.693383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767232.693385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767232.693388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066511872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767232.693391:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767232.693402:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767232.693407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767232/real 1775767232] req@ffff9d8190551880 x1862023066512000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767232.693414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066512000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767232.693420:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767232.693421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767232.693423:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767232.693426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767232.693436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767232.693441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767232/real 1775767232] req@ffff9d8190550e00 x1862023066512128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767232.693447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066512128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767232.693453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767232.693455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767232.693457:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767233.333058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767233.716956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767234.357057:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767234.740977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767235.381037:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767235.764960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767236.405070:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767236.788987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767237.429061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767237.812961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767237.813022:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767237.813028:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767237.813032:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767237.813040:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4069 00000100:00080000:0.0:1775767237.813048:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767237.813055:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767237.813089:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066512256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767237.813114:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767237.813117:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767237.813119:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767237.813121:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4069 00000100:00080000:0.0:1775767237.813124:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767237.813126:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767237.813133:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767237.813142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066512384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767237.813149:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767237.813151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767237.813153:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767237.813156:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4069 00000100:00080000:0.0:1775767237.813157:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767237.813159:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767237.813165:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767237.813165:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066512256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767237.813176:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066512512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767237.813183:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767237.813186:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767237.813187:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767237.813189:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4069 00000100:00080000:0.0:1775767237.813191:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767237.813194:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767237.813197:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767237.813206:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066512640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767237.813212:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767237.813214:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767237.813215:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767237.813217:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4069 00000100:00080000:0.0:1775767237.813219:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767237.813224:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775767237.813225:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775767237.813229:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767237.813237:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066512768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767237.813276:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767237/real 1775767237] req@ffff9d81901af100 x1862023066512256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767237.813294:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066512256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767237.813302:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767237.813305:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767237.813308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066512256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767237.813313:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767237.813329:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767237.813336:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767237/real 1775767237] req@ffff9d81901afb80 x1862023066512384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767237.813344:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066512384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767237.813349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767237.813352:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767237.813354:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767237.813370:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767237.813380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767237.813385:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767237/real 1775767237] req@ffff9d81901aed80 x1862023066512512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767237.813393:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066512512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767237.813398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767237.813399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767237.813401:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767237.813405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767237.813415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767237.813420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767237/real 1775767237] req@ffff9d81901ae680 x1862023066512640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767237.813432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066512640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767237.813439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767237.813440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767237.813442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767237.813445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767237.813455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767237.813459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767237/real 1775767237] req@ffff9d81901ad180 x1862023066512768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767237.813466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066512768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767237.813471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767237.813473:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767237.813475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066512768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767238.453040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767238.836987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767239.477049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767239.860955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767240.501086:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767240.884983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767241.525022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767241.908940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767242.549097:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767242.932956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767242.932991:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767242.932998:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767242.933002:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767242.933010:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4074 00000100:00080000:3.0:1775767242.933021:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767242.933028:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767242.933065:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066512896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767242.933098:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767242.933103:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767242.933104:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767242.933108:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4074 00000100:00080000:3.0:1775767242.933111:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767242.933116:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767242.933125:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767242.933138:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066512896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767242.933143:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066513024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767242.933166:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767242.933170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767242.933172:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767242.933176:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4074 00000100:00080000:3.0:1775767242.933188:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767242.933192:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767242.933197:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767242.933201:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767242.933219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066513152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767242.933231:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767242.933235:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767242.933237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767242.933243:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4074 00000100:00100000:0.0:1775767242.933248:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767242/real 1775767242] req@ffff9d8188741500 x1862023066512896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767242.933249:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767242.933257:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767242.933266:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767242.933279:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066512896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767242.933284:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066513280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767242.933296:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767242.933302:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767242.933305:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066512896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767242.933310:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767242.933315:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767242.933317:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767242.933324:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4074 00000100:00100000:0.0:1775767242.933325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513024:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775767242.933329:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767242.933334:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767242.933340:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767242.933354:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066513408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767242.933370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767242.933387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767242/real 1775767242] req@ffff9d8188740a80 x1862023066513024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767242.933400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066513024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767242.933406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767242.933408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767242.933410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767242.933414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767242.933466:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767242.933475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767242/real 1775767242] req@ffff9d8188742d80 x1862023066513152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767242.933483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066513152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767242.933493:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767242.933495:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767242.933497:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767242.933501:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767242.933513:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767242.933518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767242/real 1775767242] req@ffff9d8188743b80 x1862023066513280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767242.933525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066513280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767242.933532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767242.933534:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767242.933536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767242.933546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767242.933556:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767242.933561:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767242/real 1775767242] req@ffff9d8188743480 x1862023066513408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767242.933569:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066513408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767242.933574:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767242.933576:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767242.933578:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767243.573016:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767243.956944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767244.597028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767244.980984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767245.621030:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767246.005006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767246.645062:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767247.028943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767247.669045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767248.052970:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767248.052975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767248.052979:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767248.052983:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767248.052992:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4079 00000100:00080000:3.0:1775767248.053008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767248.053015:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767248.053048:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066513536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767248.053072:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767248.053075:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767248.053076:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767248.053079:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4079 00000100:00080000:3.0:1775767248.053081:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767248.053083:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767248.053091:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767248.053098:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066513664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767248.053105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767248.053108:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767248.053110:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767248.053113:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4079 00000100:00080000:3.0:1775767248.053115:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767248.053117:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767248.053120:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767248.053128:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066513536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767248.053129:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066513792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767248.053141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767248.053144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767248.053145:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767248.053149:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4079 00000100:00080000:3.0:1775767248.053150:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767248.053152:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767248.053155:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767248.053164:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066513920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767248.053169:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767248.053171:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767248.053172:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767248.053174:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4079 00000100:00080000:3.0:1775767248.053178:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767248.053180:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767248.053183:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767248.053191:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767248.053192:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066514048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767248.053242:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767248/real 1775767248] req@ffff9d8188743100 x1862023066513536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767248.053257:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066513536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767248.053265:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767248.053268:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767248.053271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066513536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767248.053276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767248.053299:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767248.053312:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767248/real 1775767248] req@ffff9d8188741180 x1862023066513664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767248.053326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066513664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767248.053335:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767248.053338:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767248.053340:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767248.053358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767248.053374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767248.053381:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767248/real 1775767248] req@ffff9d8188740e00 x1862023066513792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767248.053420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066513792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767248.053425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767248.053428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767248.053430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767248.053434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767248.053448:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767248.053455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767248/real 1775767248] req@ffff9d8188742680 x1862023066513920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767248.053463:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066513920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767248.053469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767248.053472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767248.053473:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066513920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767248.053476:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767248.053486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767248.053491:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767248/real 1775767248] req@ffff9d8188741880 x1862023066514048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767248.053503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066514048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767248.053509:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767248.053510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767248.053512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767248.693048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767249.076978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767249.717054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767250.100983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767250.741033:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767251.124986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767251.765054:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767252.149010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767252.789073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767253.172967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767253.173016:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767253.173022:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767253.173025:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767253.173033:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4084 00000100:00080000:0.0:1775767253.173041:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767253.173047:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767253.173081:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066514176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767253.173108:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767253.173110:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767253.173112:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767253.173114:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4084 00000100:00080000:0.0:1775767253.173116:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767253.173119:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767253.173126:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767253.173133:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066514304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767253.173140:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767253.173143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767253.173144:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767253.173147:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4084 00000100:00080000:0.0:1775767253.173149:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767253.173151:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767253.173154:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767253.173160:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066514432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767253.173167:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767253.173169:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066514176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767253.173170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767253.173172:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767253.173179:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4084 00000100:00080000:0.0:1775767253.173182:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767253.173184:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767253.173186:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767253.173193:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066514560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767253.173199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767253.173201:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767253.173202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767253.173204:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4084 00000100:00080000:0.0:1775767253.173207:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767253.173210:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767253.173213:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767253.173219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066514688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767253.173242:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767253.173289:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767253/real 1775767253] req@ffff9d81901ad180 x1862023066514176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767253.173317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066514176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767253.173329:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767253.173334:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767253.173354:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066514176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767253.173363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767253.173390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767253.173404:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767253/real 1775767253] req@ffff9d81901ae680 x1862023066514304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767253.173418:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066514304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767253.173427:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767253.173431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767253.173433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767253.173452:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767253.173468:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767253.173475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767253/real 1775767253] req@ffff9d81901aed80 x1862023066514432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767253.173483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066514432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767253.173488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767253.173491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767253.173493:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767253.173497:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767253.173507:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767253.173513:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767253/real 1775767253] req@ffff9d81901afb80 x1862023066514560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767253.173520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066514560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767253.173526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767253.173528:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767253.173530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767253.173534:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767253.173544:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767253.173549:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767253/real 1775767253] req@ffff9d81901af100 x1862023066514688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767253.173556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066514688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767253.173561:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767253.173563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767253.173565:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767253.813059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767254.196977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767254.837035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767255.220973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767255.861120:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767256.244988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767256.885030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767257.268950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767257.909168:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767258.292986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767258.292987:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767258.292995:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767258.292998:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767258.293006:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4089 00000100:00080000:0.0:1775767258.293015:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767258.293021:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767258.293054:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066514816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767258.293077:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767258.293080:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767258.293082:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767258.293084:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4089 00000100:00080000:0.0:1775767258.293087:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767258.293089:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767258.293097:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767258.293107:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066514944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767258.293112:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066514816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767258.293122:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767258.293127:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767258.293128:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767258.293133:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4089 00000100:00080000:0.0:1775767258.293134:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767258.293136:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767258.293140:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767258.293149:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066515072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767258.293156:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767258.293158:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767258.293160:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767258.293162:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4089 00000100:00080000:0.0:1775767258.293174:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767258.293177:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767258.293180:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767258.293188:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066515200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767258.293194:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767258.293196:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767258.293197:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767258.293200:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4089 00000100:00080000:0.0:1775767258.293203:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767258.293205:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767258.293210:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767258.293219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066515328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767258.293226:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767258.293255:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767258.293269:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767258.293284:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767258.933041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767259.317006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767259.957050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767260.341043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767260.980996:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767261.365173:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:0.0:1775767261.365220:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:0.0:1775767261.365280:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775767261.365299:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775767261.365343:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775767261.365355:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775767261.365365:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775767261.365376:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775767261.365382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767258/real 1775767261] req@ffff9d81901afb80 x1862023066514944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767261.365410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066514944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767261.365423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767261.365428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767261.365433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066514944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767261.365442:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767258/real 1775767261] req@ffff9d81901aed80 x1862023066515072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767261.365456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066515072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767261.365479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767261.365491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767261.365495:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767261.365503:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767258/real 1775767261] req@ffff9d81901ae680 x1862023066515200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767261.365516:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066515200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767261.365526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767261.365530:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767261.365533:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767261.365540:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767258/real 1775767261] req@ffff9d81901af100 x1862023066514816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767261.365553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066514816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767261.365561:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767261.365564:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767261.365567:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066514816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767261.365594:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767258/real 1775767261] req@ffff9d81901ad180 x1862023066515328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767261.365606:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066515328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767261.365616:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767261.365619:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767261.365623:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767262.005089:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767262.388955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767263.029026:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767263.412971:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767263.412979:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767263.412983:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767263.412992:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4094 00000100:00100000:3.0:1775767263.412992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767263.413004:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767263.413012:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767263.413049:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066515456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767263.413079:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767263.413084:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767263.413086:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767263.413092:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4094 00000100:00080000:1.0:1775767263.413097:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767263.413102:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767263.413108:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066515456:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775767263.413112:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767263.413127:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066515584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767263.413138:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767263.413141:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767263.413142:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767263.413146:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4094 00000100:00080000:1.0:1775767263.413158:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767263.413162:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767263.413166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767263.413167:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767263.413184:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066515712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767263.413195:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767263.413199:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767263.413201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767263.413212:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767263/real 1775767263] req@ffff9d8190550e00 x1862023066515456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767263.413213:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4094 00000100:00080000:1.0:1775767263.413222:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767263.413229:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767263.413235:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066515456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767263.413236:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:3.0:1775767263.413268:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767263.413274:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775767263.413279:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066515840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767263.413285:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066515456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767263.413301:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767263.413305:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767263.413307:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767263.413313:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4094 00000100:00080000:1.0:1775767263.413318:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767263.413323:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767263.413326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515584:192.168.202.105@tcp:38:kworker.0 00000008:00100000:1.0:1775767263.413332:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767263.413349:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066515968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767263.413366:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767263.413389:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767263/real 1775767263] req@ffff9d8190551880 x1862023066515584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767263.413415:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066515584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767263.413425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767263.413429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767263.413433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767263.413440:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767263.413480:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767263.413494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767263/real 1775767263] req@ffff9d8190551500 x1862023066515712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767263.413508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066515712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767263.413517:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767263.413521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767263.413524:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767263.413530:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767263.413554:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767263.413568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767263/real 1775767263] req@ffff9d8190553100 x1862023066515840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767263.413581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066515840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767263.413592:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767263.413595:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767263.413598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767263.413615:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767263.413636:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767263.413649:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767263/real 1775767263] req@ffff9d8190553480 x1862023066515968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767263.413668:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066515968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767263.413678:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767263.413809:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767263.413813:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066515968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767264.053048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767264.436999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767265.077054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767265.460984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767266.101085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767266.484957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767267.125028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767267.508978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767268.149071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767268.532968:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767268.532976:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767268.532980:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767268.532988:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4100 00000100:00080000:1.0:1775767268.532997:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767268.533003:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767268.533018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775767268.533039:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066516096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767268.533059:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767268.533062:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767268.533063:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767268.533066:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4100 00000100:00080000:1.0:1775767268.533068:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767268.533071:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767268.533079:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767268.533087:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066516096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767268.533090:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066516224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767268.533100:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767268.533102:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767268.533103:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767268.533106:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4100 00000100:00080000:1.0:1775767268.533109:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767268.533111:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767268.533114:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767268.533121:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066516352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767268.533127:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767268.533129:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767268.533130:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767268.533132:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4100 00000100:00080000:1.0:1775767268.533135:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767268.533137:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767268.533140:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767268.533144:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767268.533149:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066516480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767268.533157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767268.533159:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767268.533161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767268.533163:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4100 00000100:00080000:1.0:1775767268.533165:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767268.533167:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767268.533170:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767268.533179:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066516608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767268.533188:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767268/real 1775767268] req@ffff9d8190553480 x1862023066516096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767268.533223:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066516096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767268.533234:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767268.533238:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767268.533242:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066516096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767268.533264:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767268.533287:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767268.533299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767268/real 1775767268] req@ffff9d8190553100 x1862023066516224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767268.533312:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066516224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767268.533321:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767268.533325:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767268.533327:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767268.533333:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767268.533352:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767268.533362:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767268/real 1775767268] req@ffff9d8190551500 x1862023066516352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767268.533374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066516352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767268.533383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767268.533386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767268.533389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767268.533395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767268.533412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767268.533419:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767268/real 1775767268] req@ffff9d8190551880 x1862023066516480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767268.533427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066516480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767268.533437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767268.533439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767268.533441:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767268.533445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767268.533455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767268.533460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767268/real 1775767268] req@ffff9d8190550e00 x1862023066516608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767268.533467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066516608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767268.533472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767268.533474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767268.533475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767269.173062:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767269.556988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767270.197088:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767270.581040:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767271.221073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767271.605009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767272.245116:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767272.628996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767273.269062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767273.652959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767273.653011:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767273.653015:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767273.653019:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767273.653028:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4105 00000100:00080000:3.0:1775767273.653037:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767273.653043:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767273.653077:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066516736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767273.653104:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767273.653108:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767273.653111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767273.653116:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4105 00000100:00080000:3.0:1775767273.653120:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767273.653125:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767273.653131:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066516736:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775767273.653134:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767273.653164:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066516864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767273.653177:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767273.653181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767273.653183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767273.653197:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4105 00000800:00000100:0.0:1775767273.653200:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767273.653205:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767273.653210:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767273.653215:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767273.653227:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066516992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767273.653238:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767273/real 1775767273] req@ffff9d8188743800 x1862023066516736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767273.653241:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767273.653249:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767273.653253:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767273.653258:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4105 00000100:00080000:3.0:1775767273.653261:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767273.653264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066516736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767273.653266:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767273.653273:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775767273.653281:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767273.653287:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767273.653289:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066517120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767273.653294:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066516736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767273.653303:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767273.653305:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767273.653306:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767273.653309:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4105 00000100:00080000:3.0:1775767273.653311:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767273.653313:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767273.653318:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767273.653319:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767273.653336:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066517248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767273.653352:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767273.653364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767273/real 1775767273] req@ffff9d8188741f80 x1862023066516864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767273.653374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066516864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767273.653380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767273.653382:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767273.653385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767273.653389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767273.653401:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767273.653407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767273/real 1775767273] req@ffff9d8188742300 x1862023066516992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767273.653414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066516992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767273.653419:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767273.653421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767273.653423:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066516992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767273.653426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767273.653436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767273.653442:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767273/real 1775767273] req@ffff9d8188740000 x1862023066517120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767273.653450:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066517120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767273.653456:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767273.653458:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767273.653460:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767273.653469:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767273.653479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767273.653484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767273/real 1775767273] req@ffff9d8188740700 x1862023066517248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767273.653492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066517248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767273.653497:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767273.653499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767273.653501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767274.293068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767274.677023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767275.317081:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767275.701082:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767276.341038:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767276.724987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767277.365049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767277.748970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767278.389052:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767278.772955:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767278.772966:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767278.772971:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767278.772992:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4110 00000100:00080000:3.0:1775767278.773004:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775767278.773007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767278.773013:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767278.773058:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066517376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767278.773091:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767278.773096:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767278.773110:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767278.773115:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4110 00000100:00080000:3.0:1775767278.773119:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767278.773123:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767278.773132:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767278.773147:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066517504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767278.773149:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066517376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767278.773164:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767278.773167:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767278.773168:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767278.773171:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4110 00000100:00080000:3.0:1775767278.773174:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767278.773176:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767278.773179:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767278.773188:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066517632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767278.773199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767278.773203:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767278.773205:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767278.773209:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4110 00000100:00080000:3.0:1775767278.773212:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767278.773216:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767278.773222:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767278.773225:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767278.773241:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066517760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767278.773251:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767278.773254:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767278.773256:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767278.773260:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4110 00000100:00080000:3.0:1775767278.773265:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767278.773269:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767278.773274:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767278.773276:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767278/real 1775767278] req@ffff9d8188740380 x1862023066517376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767278.773287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066517888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767278.773328:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066517376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767278.773341:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767278.773346:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767278.773350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066517376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767278.773373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767278.773399:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767278.773414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767278/real 1775767278] req@ffff9d8188741c00 x1862023066517504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767278.773427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066517504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767278.773437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767278.773441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767278.773444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767278.773451:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767278.773473:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767278.773485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767278/real 1775767278] req@ffff9d8188742a00 x1862023066517632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767278.773498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066517632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767278.773514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767278.773517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767278.773521:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767278.773528:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767278.773550:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767278.773562:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767278/real 1775767278] req@ffff9d8188740700 x1862023066517760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767278.773575:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066517760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767278.773585:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767278.773588:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767278.773592:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767278.773598:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767278.773620:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767278.773632:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767278/real 1775767278] req@ffff9d8188740000 x1862023066517888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767278.773645:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066517888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767278.773654:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767278.773657:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767278.773660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066517888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767279.413015:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767279.797033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767280.437059:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767280.820997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767281.461128:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767281.844989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767282.485060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767282.869003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767283.509160:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767283.892962:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767283.892973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767283.892978:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767283.892985:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767283.892997:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4115 00000100:00080000:3.0:1775767283.893012:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767283.893022:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767283.893080:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066518016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767283.893110:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767283.893114:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767283.893116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767283.893121:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4115 00000100:00080000:3.0:1775767283.893125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767283.893130:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767283.893143:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767283.893159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066518144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767283.893162:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066518016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767283.893180:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767283.893184:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767283.893186:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767283.893192:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4115 00000100:00080000:3.0:1775767283.893196:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767283.893200:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767283.893206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767283.893219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066518272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767283.893230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767283.893234:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767283.893236:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767283.893240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4115 00000100:00080000:3.0:1775767283.893244:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767283.893248:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767283.893253:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767283.893257:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767283.893270:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066518400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767283.893279:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767283.893282:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767283.893284:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767283.893287:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4115 00000100:00080000:3.0:1775767283.893290:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767283.893292:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767283.893296:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767283.893306:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066518528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767283.893323:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767283/real 1775767283] req@ffff9d8188742300 x1862023066518016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767283.893343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066518016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767283.893361:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767283.893365:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767283.893369:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066518016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767283.893392:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767283.893412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767283.893420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767283/real 1775767283] req@ffff9d8188741f80 x1862023066518144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767283.893429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066518144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767283.893435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767283.893438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767283.893440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767283.893443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767283.893466:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767283.893472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767283/real 1775767283] req@ffff9d8188743800 x1862023066518272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767283.893481:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066518272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767283.893486:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767283.893488:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767283.893490:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767283.893494:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767283.893509:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767283.893518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767283/real 1775767283] req@ffff9d8188741880 x1862023066518400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767283.893540:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066518400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767283.893551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767283.893555:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767283.893559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767283.893565:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767283.893583:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767283.893590:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767283/real 1775767283] req@ffff9d8188742680 x1862023066518528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767283.893604:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066518528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767283.893610:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767283.893613:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767283.893615:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767284.533090:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767284.917016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767285.557085:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767285.940952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767286.581073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767286.964991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767287.605076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767287.989006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767288.629049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767289.012992:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767289.013000:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767289.013014:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767289.013025:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4120 00000100:00080000:1.0:1775767289.013040:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767289.013048:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767289.013066:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775767289.013086:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066518656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767289.013111:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767289.013115:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767289.013117:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767289.013120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4120 00000100:00080000:1.0:1775767289.013125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767289.013129:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767289.013137:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767289.013148:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066518656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767289.013151:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066518784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767289.013162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767289.013164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767289.013165:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767289.013167:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4120 00000100:00080000:1.0:1775767289.013169:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767289.013172:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767289.013175:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767289.013183:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066518912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767289.013189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767289.013191:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767289.013193:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767289.013195:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4120 00000100:00080000:1.0:1775767289.013205:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767289.013208:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767289.013212:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767289.013216:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767289.013224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066519040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767289.013236:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767289.013238:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767289.013239:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767289.013241:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4120 00000100:00080000:1.0:1775767289.013243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767289.013252:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767289.013255:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767289.013267:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066519168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767289.013281:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767289/real 1775767289] req@ffff9d8190550e00 x1862023066518656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767289.013297:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066518656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767289.013305:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767289.013308:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767289.013311:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066518656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767289.013329:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767289.013346:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767289.013355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767289/real 1775767289] req@ffff9d8190551880 x1862023066518784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767289.013363:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066518784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767289.013369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767289.013371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767289.013374:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767289.013377:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767289.013387:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767289.013393:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767289/real 1775767289] req@ffff9d8190551500 x1862023066518912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767289.013400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066518912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767289.013405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767289.013407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767289.013408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066518912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767289.013412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767289.013422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767289.013427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767289/real 1775767289] req@ffff9d8190553100 x1862023066519040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767289.013435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066519040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767289.013441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767289.013442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767289.013445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767289.013448:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767289.013461:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767289.013471:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767289/real 1775767289] req@ffff9d8190553480 x1862023066519168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767289.013483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066519168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767289.013492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767289.013496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767289.013499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767289.653029:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767290.036985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767290.677033:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767291.061034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767291.701062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767292.085003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767292.725106:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767293.108953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767293.749061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767294.132982:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767294.132985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767294.132994:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767294.132999:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767294.133009:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4125 00000100:00080000:3.0:1775767294.133020:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767294.133029:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767294.133067:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066519296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767294.133095:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767294.133099:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767294.133101:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767294.133105:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4125 00000100:00080000:3.0:1775767294.133109:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767294.133113:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767294.133124:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767294.133139:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066519424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767294.133144:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066519296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767294.133160:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767294.133164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767294.133167:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767294.133172:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4125 00000100:00080000:3.0:1775767294.133177:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767294.133181:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767294.133187:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767294.133202:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767294.133203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066519552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767294.133217:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767294.133221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767294.133223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767294.133228:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4125 00000100:00080000:3.0:1775767294.133233:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767294.133237:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767294.133242:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767294/real 1775767294] req@ffff9d8188740e00 x1862023066519296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767294.133245:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767294.133263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066519296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767294.133266:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066519680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767294.133279:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767294.133284:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775767294.133285:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767294.133289:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066519296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767294.133292:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767294.133295:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767294.133301:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4125 00000100:00080000:3.0:1775767294.133305:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767294.133310:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767294.133311:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519424:192.168.202.105@tcp:38:kworker.0 00000008:00100000:3.0:1775767294.133319:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767294.133336:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066519808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767294.133347:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767294.133360:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767294/real 1775767294] req@ffff9d8188741180 x1862023066519424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767294.133369:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066519424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767294.133375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767294.133378:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767294.133380:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767294.133384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767294.133395:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767294.133402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767294/real 1775767294] req@ffff9d8188743100 x1862023066519552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767294.133415:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066519552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767294.133421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767294.133423:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767294.133425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767294.133428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767294.133439:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767294.133445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767294/real 1775767294] req@ffff9d8188743480 x1862023066519680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767294.133457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066519680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767294.133464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767294.133466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767294.133468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767294.133476:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767294.133486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767294.133492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767294/real 1775767294] req@ffff9d8188743b80 x1862023066519808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767294.133499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066519808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767294.133505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767294.133506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767294.133508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066519808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767294.773072:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767295.157006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767295.797048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767296.180972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767296.821028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767297.204998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767297.845000:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767298.228991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767298.869019:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767299.253029:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775767299.253035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767299.253037:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767299.253042:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767299.253050:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4130 00000100:00080000:2.0:1775767299.253058:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767299.253064:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767299.253100:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066519936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767299.253130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767299.253134:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767299.253136:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767299.253139:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4130 00000100:00080000:2.0:1775767299.253143:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767299.253148:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767299.253158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767299.253170:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066520064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767299.253179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767299.253181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767299.253183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767299.253188:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066519936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767299.253189:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4130 00000100:00080000:2.0:1775767299.253194:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767299.253200:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767299.253204:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767299.253213:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066520192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767299.253219:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767299.253222:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767299.253224:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767299.253226:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4130 00000100:00080000:2.0:1775767299.253243:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767299.253248:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767299.253253:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767299.253256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767299.253272:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066520320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767299.253283:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767299.253287:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767299.253289:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767299.253295:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4130 00000100:00080000:2.0:1775767299.253299:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767299.253305:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767299/real 1775767299] req@ffff9d81984a1180 x1862023066519936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767299.253307:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767299.253317:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767299.253333:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066519936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767299.253335:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066520448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767299.253360:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767299.253366:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767299.253371:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066519936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767299.253395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767299.253418:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767299.253432:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767299/real 1775767299] req@ffff9d81984a2680 x1862023066520064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767299.253446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066520064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767299.253456:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767299.253459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767299.253463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767299.253470:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767299.253491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767299.253502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767299/real 1775767299] req@ffff9d81984a0000 x1862023066520192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767299.253516:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066520192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767299.253525:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767299.253529:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767299.253533:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767299.253539:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767299.253557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767299.253565:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767299/real 1775767299] req@ffff9d81984a3b80 x1862023066520320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767299.253577:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066520320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767299.253587:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767299.253590:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767299.253594:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767299.253599:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767299.253619:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767299.253630:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767299/real 1775767299] req@ffff9d81984a1f80 x1862023066520448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767299.253643:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066520448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767299.253652:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767299.253655:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767299.253658:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767299.893060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767300.277003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767300.917045:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767301.300987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767301.941101:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767302.324999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767302.965123:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767303.348959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767303.989127:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767304.373022:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767304.373030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767304.373035:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767304.373045:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767304.373058:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4135 00000100:00080000:1.0:1775767304.373071:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767304.373079:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767304.373129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066520576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767304.373157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767304.373160:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767304.373162:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767304.373165:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4135 00000100:00080000:1.0:1775767304.373168:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767304.373170:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767304.373180:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767304.373189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066520704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767304.373199:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767304.373203:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767304.373207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767304.373208:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066520576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767304.373215:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4135 00000100:00080000:1.0:1775767304.373219:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767304.373224:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767304.373231:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767304.373245:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066520832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767304.373257:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767304.373261:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767304.373263:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767304.373267:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4135 00000100:00080000:1.0:1775767304.373272:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767304.373275:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767304.373279:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767304.373284:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767304.373299:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066520960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767304.373307:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767304.373310:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767304.373311:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767304.373314:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4135 00000100:00080000:1.0:1775767304.373317:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767304.373319:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767304.373323:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767304.373330:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066521088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767304.373355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767304/real 1775767304] req@ffff9d8190553480 x1862023066520576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767304.373379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066520576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767304.373389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767304.373393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767304.373396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066520576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767304.373420:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767304.373440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767304.373450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767304/real 1775767304] req@ffff9d8190553100 x1862023066520704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767304.373459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066520704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767304.373465:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767304.373468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767304.373472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767304.373479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767304.373500:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767304.373511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767304/real 1775767304] req@ffff9d8190551500 x1862023066520832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767304.373544:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066520832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767304.373553:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767304.373557:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767304.373561:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767304.373568:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767304.373591:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767304.373600:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767304/real 1775767304] req@ffff9d8190551880 x1862023066520960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767304.373609:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066520960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767304.373616:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767304.373619:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767304.373621:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066520960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767304.373625:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767304.373637:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767304.373643:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767304/real 1775767304] req@ffff9d8190550e00 x1862023066521088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767304.373650:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066521088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767304.373661:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767304.373663:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767304.373665:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767305.013063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767305.396951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767306.037100:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767306.421000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767307.061079:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767307.445003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767308.085091:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767308.469011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767309.109048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767309.492968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767309.492975:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767309.492983:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767309.492987:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767309.492998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4141 00000100:00080000:1.0:1775767309.493008:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767309.493015:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767309.493048:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066521216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767309.493077:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767309.493081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767309.493083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767309.493087:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4141 00000100:00080000:1.0:1775767309.493091:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767309.493095:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767309.493104:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767309.493117:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066521344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767309.493130:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066521216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767309.493130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767309.493139:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767309.493141:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767309.493146:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4141 00000100:00080000:1.0:1775767309.493149:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767309.493152:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767309.493157:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767309.493169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066521472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767309.493180:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767309.493183:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767309.493185:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767309.493189:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4141 00000100:00080000:1.0:1775767309.493193:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767309.493199:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767309.493199:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767309.493207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767309.493220:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066521600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767309.493231:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767309.493235:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767309.493237:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767309.493241:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4141 00000100:00100000:0.0:1775767309.493245:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767309/real 1775767309] req@ffff9d8190550e00 x1862023066521216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767309.493247:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767309.493255:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767309.493261:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767309.493267:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066521216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767309.493274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066521728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767309.493286:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767309.493294:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767309.493300:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066521216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767309.493324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767309.493344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767309.493353:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767309/real 1775767309] req@ffff9d8190551880 x1862023066521344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767309.493362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066521344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767309.493368:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767309.493370:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767309.493372:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767309.493378:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767309.493397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767309.493407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767309/real 1775767309] req@ffff9d8190551500 x1862023066521472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767309.493421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066521472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767309.493430:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767309.493433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767309.493436:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767309.493442:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767309.493467:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767309.493475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767309/real 1775767309] req@ffff9d8190553100 x1862023066521600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767309.493483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066521600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767309.493489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767309.493491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767309.493494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767309.493497:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767309.493510:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767309.493515:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767309/real 1775767309] req@ffff9d8190553480 x1862023066521728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767309.493522:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066521728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767309.493527:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767309.493529:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767309.493531:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767310.133022:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767310.516977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767311.157207:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767311.540958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767312.181024:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767312.564983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767313.205016:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767313.588952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767314.229088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767314.612984:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767314.612987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767314.612995:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767314.612999:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767314.613007:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4146 00000100:00080000:3.0:1775767314.613017:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767314.613022:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767314.613054:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066521856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767314.613077:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767314.613081:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767314.613083:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767314.613087:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4146 00000100:00080000:3.0:1775767314.613091:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767314.613095:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767314.613106:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767314.613121:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066521984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767314.613126:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066521856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767314.613142:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767314.613146:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767314.613148:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767314.613154:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4146 00000100:00080000:3.0:1775767314.613159:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767314.613162:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767314.613166:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767314.613177:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066522112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767314.613184:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767314.613186:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767314.613188:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767314.613191:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4146 00000800:00000100:0.0:1775767314.613196:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767314.613197:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767314.613202:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767314.613206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767314.613216:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066522240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767314.613228:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767314.613230:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767314.613232:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767314.613235:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4146 00000100:00080000:3.0:1775767314.613237:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767314.613240:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767314/real 1775767314] req@ffff9d8188742d80 x1862023066521856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767314.613243:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767314.613249:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767314.613261:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066522368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767314.613264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066521856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767314.613287:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767314.613291:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767314.613294:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066521856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767314.613314:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767314.613331:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767314.613339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767314/real 1775767314] req@ffff9d8188740a80 x1862023066521984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767314.613353:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066521984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767314.613358:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767314.613360:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767314.613362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066521984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767314.613366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767314.613377:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767314.613382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767314/real 1775767314] req@ffff9d8188741500 x1862023066522112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767314.613389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066522112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767314.613394:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767314.613396:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767314.613398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767314.613401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767314.613412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767314.613417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767314/real 1775767314] req@ffff9d8188743b80 x1862023066522240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767314.613424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066522240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767314.613434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767314.613436:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767314.613438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767314.613441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767314.613460:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767314.613466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767314/real 1775767314] req@ffff9d8188743480 x1862023066522368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767314.613473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066522368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767314.613478:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767314.613480:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767314.613482:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767315.253037:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767315.637022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767316.277054:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767316.660973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767317.301021:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767317.684940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767318.325074:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767318.708973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767319.349067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767319.732980:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767319.732984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767319.732990:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767319.732994:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767319.733002:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4151 00000100:00080000:3.0:1775767319.733011:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767319.733017:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767319.733058:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066522496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767319.733081:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767319.733085:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767319.733086:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767319.733092:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4151 00000100:00080000:3.0:1775767319.733096:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767319.733100:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767319.733109:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767319.733121:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066522624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767319.733132:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767319.733134:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767319.733136:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767319.733140:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4151 00000100:00100000:0.0:1775767319.733141:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066522496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767319.733143:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767319.733146:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767319.733150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767319.733159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066522752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767319.733168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767319.733171:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767319.733172:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767319.733176:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4151 00000100:00080000:3.0:1775767319.733195:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767319.733198:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767319.733200:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767319.733205:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767319.733214:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066522880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767319.733224:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767319.733227:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767319.733228:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767319.733232:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4151 00000100:00080000:3.0:1775767319.733235:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767319.733237:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767319.733241:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767319/real 1775767319] req@ffff9d8188743100 x1862023066522496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767319.733241:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767319.733249:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066523008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767319.733258:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066522496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767319.733268:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767319.733272:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767319.733276:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066522496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767319.733283:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767319.733305:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767319.733315:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767319/real 1775767319] req@ffff9d8188741180 x1862023066522624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767319.733323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066522624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767319.733329:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767319.733331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767319.733333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767319.733353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767319.733372:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767319.733383:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767319/real 1775767319] req@ffff9d8188740e00 x1862023066522752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767319.733396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066522752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767319.733405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767319.733408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767319.733411:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767319.733417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767319.733433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767319.733440:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767319/real 1775767319] req@ffff9d8188742680 x1862023066522880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767319.733447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066522880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767319.733454:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767319.733456:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767319.733458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066522880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767319.733462:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767319.733471:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767319.733476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767319/real 1775767319] req@ffff9d8188741880 x1862023066523008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767319.733483:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066523008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767319.733489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767319.733490:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767319.733492:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767320.373071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767320.756983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767321.397051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767321.780988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767322.421055:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767322.804988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767323.445043:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767323.828951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767324.469040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767324.852939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767324.852964:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767324.852970:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767324.852973:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767324.852981:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4156 00000100:00080000:0.0:1775767324.852990:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767324.852995:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767324.853028:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066523136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767324.853051:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767324.853054:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767324.853055:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767324.853058:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4156 00000100:00080000:0.0:1775767324.853061:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767324.853063:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767324.853071:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767324.853082:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066523264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767324.853083:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066523136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767324.853097:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767324.853099:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767324.853100:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767324.853103:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4156 00000100:00080000:0.0:1775767324.853105:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767324.853107:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767324.853111:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767324.853122:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066523392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767324.853137:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767324.853141:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767324.853143:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767324.853147:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4156 00000100:00080000:0.0:1775767324.853151:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767324.853154:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767324.853159:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767324.853174:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066523520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767324.853183:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523264:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775767324.853185:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767324.853190:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767324.853192:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767324.853197:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4156 00000100:00080000:0.0:1775767324.853205:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767324.853210:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523392:192.168.202.105@tcp:38:kworker.0 00000100:00080000:0.0:1775767324.853211:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767324.853222:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767324.853236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066523648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767324.853237:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767324.853266:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767325.493063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767325.877002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767326.517041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767326.900948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767327.541022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767327.924972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767327.925107:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767327.925140:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767327.925154:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767327.925187:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767327.925203:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767327.925211:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767327.925219:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775767327.925256:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767324/real 1775767327] req@ffff9d81901afb80 x1862023066523648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767327.925290:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066523648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767327.925301:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767327.925306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767327.925310:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523648:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767327.925317:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767324/real 1775767327] req@ffff9d81901aed80 x1862023066523520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767327.925329:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066523520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767327.925339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767327.925343:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767327.925346:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767327.925353:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767324/real 1775767327] req@ffff9d81901af100 x1862023066523264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767327.925365:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066523264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767327.925373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767327.925376:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767327.925380:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767327.925385:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767324/real 1775767327] req@ffff9d81901ae680 x1862023066523392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767327.925397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066523392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767327.925404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767327.925407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767327.925410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767327.925416:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767324/real 1775767327] req@ffff9d81901ad180 x1862023066523136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767327.925428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066523136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767327.925443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767327.925446:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767327.925449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066523136:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775767328.565067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767328.948996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767329.588991:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767329.972976:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767329.972978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767329.972985:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767329.972990:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767329.972998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4161 00000100:00080000:0.0:1775767329.973009:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767329.973015:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767329.973045:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066523776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767329.973068:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767329.973071:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767329.973072:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767329.973075:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4161 00000100:00080000:0.0:1775767329.973078:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767329.973080:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767329.973086:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767329.973094:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066523904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767329.973101:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767329.973104:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767329.973105:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767329.973109:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4161 00000100:00080000:0.0:1775767329.973111:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767329.973117:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767329.973117:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066523776:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775767329.973125:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767329.973138:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066524032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767329.973148:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767329.973152:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767329.973154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767329.973159:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4161 00000100:00080000:0.0:1775767329.973182:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767329.973186:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775767329.973193:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775767329.973203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767329.973223:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066524160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767329.973234:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767329.973239:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767329/real 1775767329] req@ffff9d81901ad180 x1862023066523776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767329.973240:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767329.973244:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767329.973255:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4161 00000100:00080000:0.0:1775767329.973261:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767329.973266:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767329.973270:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066523776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767329.973273:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775767329.973289:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1775767329.973294:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066524288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767329.973299:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767329.973308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066523776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767329.973321:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767329.973347:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767329.973361:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767329/real 1775767329] req@ffff9d81901ae680 x1862023066523904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767329.973375:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066523904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767329.973384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767329.973387:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767329.973391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066523904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767329.973417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767329.973440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767329.973452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767329/real 1775767329] req@ffff9d81901af100 x1862023066524032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767329.973465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066524032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767329.973473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767329.973477:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767329.973480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767329.973486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767329.973513:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767329.973529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767329/real 1775767329] req@ffff9d81901aed80 x1862023066524160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767329.973542:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066524160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767329.973551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767329.973554:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767329.973557:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767329.973563:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767329.973583:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767329.973594:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767329/real 1775767329] req@ffff9d81901afb80 x1862023066524288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767329.973606:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066524288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767329.973616:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767329.973633:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767329.973637:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767330.613040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767330.996968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767331.637070:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767332.020989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767332.661041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767333.044957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767333.685061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767334.069012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767334.709047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767335.093008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767335.093012:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767335.093021:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767335.093026:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767335.093036:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4166 00000100:00080000:1.0:1775767335.093049:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767335.093057:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767335.093104:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066524416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767335.093132:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767335.093135:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767335.093136:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767335.093140:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4166 00000100:00080000:1.0:1775767335.093142:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767335.093145:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767335.093155:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767335.093167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066524544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767335.093176:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767335.093179:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066524416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767335.093180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767335.093183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767335.093192:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4166 00000100:00080000:1.0:1775767335.093194:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767335.093196:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767335.093199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767335.093208:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066524672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767335.093214:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767335.093217:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767335.093218:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767335.093222:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4166 00000100:00080000:1.0:1775767335.093225:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767335.093228:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767335.093231:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767335.093238:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066524800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767335.093244:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767335.093247:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767335.093248:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767335.093251:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4166 00000800:00000100:3.0:1775767335.093261:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767335.093262:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767335.093269:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767335.093273:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767335.093282:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066524928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767335.093320:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767335/real 1775767335] req@ffff9d8190553480 x1862023066524416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767335.093337:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066524416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767335.093346:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767335.093350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767335.093353:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066524416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767335.093375:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767335.093393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767335.093401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767335/real 1775767335] req@ffff9d8190553100 x1862023066524544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767335.093410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066524544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767335.093416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767335.093419:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767335.093421:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767335.093424:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767335.093434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767335.093440:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767335/real 1775767335] req@ffff9d8190551500 x1862023066524672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767335.093447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066524672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767335.093458:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767335.093460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767335.093462:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767335.093465:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767335.093476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767335.093481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767335/real 1775767335] req@ffff9d8190551880 x1862023066524800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767335.093488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066524800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767335.093501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767335.093503:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767335.093506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767335.093510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767335.093521:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767335.093527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767335/real 1775767335] req@ffff9d8190550e00 x1862023066524928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767335.093534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066524928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767335.093540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767335.093542:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767335.093545:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066524928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767335.733095:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767336.117024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767336.757029:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767337.140929:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767337.781082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767338.164960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767338.805031:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767339.188961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767339.829010:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767340.212948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767340.212950:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767340.212957:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767340.212961:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767340.212969:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4171 00000100:00080000:1.0:1775767340.212977:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767340.212983:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767340.213024:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066525056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767340.213049:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767340.213052:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767340.213053:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767340.213055:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4171 00000100:00080000:1.0:1775767340.213057:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767340.213059:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767340.213066:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767340.213076:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066525184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767340.213084:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767340.213087:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767340.213088:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767340.213092:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4171 00000100:00100000:3.0:1775767340.213092:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066525056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767340.213096:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767340.213102:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767340.213105:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767340.213115:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066525312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767340.213121:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767340.213123:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767340.213125:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767340.213127:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4171 00000100:00080000:1.0:1775767340.213141:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767340.213144:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767340.213148:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767340.213149:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767340.213163:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066525440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767340.213173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767340.213176:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767340.213178:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767340.213183:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4171 00000100:00080000:1.0:1775767340.213188:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767340.213193:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767340.213198:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767340/real 1775767340] req@ffff9d8190550e00 x1862023066525056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767340.213200:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767340.213218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066525568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767340.213225:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066525056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767340.213238:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767340.213242:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767340.213245:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066525056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767340.213264:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767340.213281:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767340.213290:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767340/real 1775767340] req@ffff9d8190551880 x1862023066525184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767340.213298:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066525184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767340.213303:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767340.213306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767340.213308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767340.213313:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767340.213324:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767340.213330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767340/real 1775767340] req@ffff9d8190551500 x1862023066525312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767340.213337:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066525312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767340.213342:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767340.213344:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767340.213346:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767340.213349:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767340.213359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767340.213365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767340/real 1775767340] req@ffff9d8190553100 x1862023066525440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767340.213371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066525440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767340.213377:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767340.213379:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767340.213381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767340.213384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767340.213393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767340.213398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767340/real 1775767340] req@ffff9d8190553480 x1862023066525568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767340.213406:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066525568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767340.213411:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767340.213413:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767340.213414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767340.853054:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767341.237014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767341.877060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767342.260960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767342.901092:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767343.285012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767343.925060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767344.308973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767344.949065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767345.333049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767345.333095:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767345.333105:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767345.333111:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767345.333139:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4176 00000100:00080000:0.0:1775767345.333153:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767345.333160:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767345.333212:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066525696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767345.333241:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767345.333244:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767345.333246:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767345.333249:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4176 00000100:00080000:0.0:1775767345.333252:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767345.333255:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767345.333266:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767345.333277:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066525824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767345.333285:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767345.333289:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767345.333290:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767345.333295:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4176 00000100:00080000:0.0:1775767345.333298:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767345.333301:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767345.333306:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767345.333315:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066525696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767345.333318:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066525952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767345.333353:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767345.333358:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767345.333360:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767345.333368:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4176 00000100:00080000:0.0:1775767345.333374:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767345.333379:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767345.333387:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767345.333403:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066526080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767345.333413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775767345.333448:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767345.333452:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767345.333454:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767345.333458:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4176 00000100:00080000:0.0:1775767345.333462:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767345.333467:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767345.333474:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767345.333502:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066526208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767345.333521:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767345/real 1775767345] req@ffff9d81901afb80 x1862023066525696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767345.333558:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066525696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767345.333571:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767345.333577:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767345.333581:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066525696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767345.333589:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767345.333627:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767345.333640:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767345/real 1775767345] req@ffff9d81901aed80 x1862023066525824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767345.333661:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066525824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767345.333669:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767345.333672:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767345.333676:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767345.333699:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767345.333720:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767345.333731:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767345/real 1775767345] req@ffff9d81901af100 x1862023066525952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767345.333744:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066525952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767345.333753:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767345.333756:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767345.333770:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066525952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767345.333777:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767345.333808:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767345.333821:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767345/real 1775767345] req@ffff9d81901ae680 x1862023066526080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767345.333837:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066526080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767345.333854:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767345.333859:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767345.333862:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767345.333869:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767345.333921:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767345.333933:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767345/real 1775767345] req@ffff9d81901ad180 x1862023066526208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767345.333945:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066526208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767345.333954:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767345.333969:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767345.333972:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767345.973028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767346.356972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767346.997124:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767347.381048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767348.021054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767348.404972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767349.045062:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767349.428960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767350.069077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767350.452998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767350.453037:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767350.453047:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767350.453051:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767350.453062:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4181 00000100:00080000:1.0:1775767350.453074:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767350.453082:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767350.453130:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066526336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767350.453159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767350.453164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767350.453166:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767350.453168:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4181 00000100:00080000:1.0:1775767350.453171:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767350.453173:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767350.453183:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767350.453196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066526464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767350.453206:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767350.453209:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767350.453213:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767350.453215:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066526336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767350.453223:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4181 00000100:00080000:1.0:1775767350.453229:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767350.453233:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767350.453238:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767350.453250:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066526592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767350.453257:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767350.453260:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767350.453262:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767350.453265:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4181 00000100:00080000:1.0:1775767350.453282:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767350.453285:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767350.453290:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767350.453298:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767350.453305:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066526720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767350.453315:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767350.453318:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767350.453320:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767350.453323:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4181 00000100:00080000:1.0:1775767350.453325:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767350.453328:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767350.453331:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767350.453338:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066526848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767350.453359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767350/real 1775767350] req@ffff9d8190553480 x1862023066526336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767350.453382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066526336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767350.453401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767350.453406:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767350.453411:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066526336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767350.453462:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767350.453488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767350.453501:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767350/real 1775767350] req@ffff9d8190553100 x1862023066526464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767350.453515:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066526464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767350.453524:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767350.453528:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767350.453531:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767350.453538:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767350.453558:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767350.453568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767350/real 1775767350] req@ffff9d8190551500 x1862023066526592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767350.453581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066526592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767350.453590:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767350.453594:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767350.453597:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767350.453605:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767350.453624:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767350.453635:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767350/real 1775767350] req@ffff9d8190551880 x1862023066526720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767350.453647:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066526720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767350.453658:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767350.453661:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767350.453665:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767350.453672:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767350.453689:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767350.453698:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767350/real 1775767350] req@ffff9d8190550e00 x1862023066526848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767350.453711:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066526848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767350.453720:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767350.453724:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767350.453727:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066526848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767351.093059:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767351.476976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767352.117064:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767352.501084:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767353.141115:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767353.524985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767354.165034:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767354.548957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767355.189087:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767355.572974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767355.573030:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767355.573036:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767355.573039:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767355.573047:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4187 00000100:00080000:0.0:1775767355.573056:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767355.573061:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767355.573096:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066526976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767355.573119:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767355.573122:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767355.573123:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767355.573126:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4187 00000100:00080000:0.0:1775767355.573128:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767355.573131:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767355.573138:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767355.573146:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066527104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767355.573154:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767355.573157:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767355.573158:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767355.573163:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066526976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767355.573164:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4187 00000100:00080000:0.0:1775767355.573168:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767355.573172:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767355.573175:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767355.573183:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066527232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767355.573190:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767355.573193:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767355.573194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767355.573196:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4187 00000100:00080000:0.0:1775767355.573211:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767355.573214:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767355.573216:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767355.573221:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767355.573226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066527360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767355.573233:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767355.573236:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767355.573237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767355.573240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4187 00000100:00080000:0.0:1775767355.573242:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767355.573244:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767355.573247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767355.573255:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066527488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767355.573265:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767355/real 1775767355] req@ffff9d81901ad180 x1862023066526976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767355.573281:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066526976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767355.573288:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767355.573291:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767355.573294:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066526976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767355.573310:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767355.573323:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767355.573330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767355/real 1775767355] req@ffff9d81901ae680 x1862023066527104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767355.573338:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066527104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767355.573343:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767355.573345:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767355.573348:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767355.573351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767355.573360:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767355.573365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767355/real 1775767355] req@ffff9d81901af100 x1862023066527232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767355.573373:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066527232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767355.573378:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767355.573380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767355.573397:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767355.573401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767355.573413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767355.573420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767355/real 1775767355] req@ffff9d81901aed80 x1862023066527360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767355.573427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066527360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767355.573434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767355.573436:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767355.573438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767355.573441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767355.573450:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767355.573455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767355/real 1775767355] req@ffff9d81901afb80 x1862023066527488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767355.573462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066527488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767355.573471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767355.573474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767355.573475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767356.213026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767356.597015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767357.237092:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767357.620968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767358.261074:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767358.645003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767359.285105:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767359.668991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767360.309060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767360.693051:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767360.693055:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767360.693069:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767360.693076:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767360.693087:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4192 00000100:00080000:3.0:1775767360.693101:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767360.693110:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767360.693166:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066527616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767360.693202:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767360.693206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767360.693207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767360.693210:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4192 00000100:00080000:3.0:1775767360.693213:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767360.693215:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767360.693226:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767360.693236:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066527744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767360.693247:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767360.693250:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767360.693252:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767360.693256:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4192 00000100:00080000:3.0:1775767360.693260:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767360.693262:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767360.693265:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767360.693274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066527872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767360.693284:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767360.693292:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775767360.693294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066527616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767360.693295:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767360.693315:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4192 00000100:00080000:3.0:1775767360.693318:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767360.693320:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767360.693324:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767360.693355:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066528000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767360.693362:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767360.693364:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767360.693366:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767360.693370:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4192 00000100:00080000:3.0:1775767360.693374:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767360.693378:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767360.693383:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767360.693394:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066528128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767360.693443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767360.693538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767360/real 1775767360] req@ffff9d8188743800 x1862023066527616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767360.693579:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066527616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767360.693602:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767360.693612:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767360.693620:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066527616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767360.693635:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767360.693684:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767360.693710:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767360/real 1775767360] req@ffff9d8188741f80 x1862023066527744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767360.693730:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066527744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767360.693741:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767360.693745:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767360.693750:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767360.693780:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767360.693810:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767360.693823:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767360/real 1775767360] req@ffff9d8188742300 x1862023066527872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767360.693838:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066527872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767360.693856:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767360.693861:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767360.693864:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066527872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767360.693871:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767360.693933:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767360.693946:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767360/real 1775767360] req@ffff9d8188740000 x1862023066528000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767360.693961:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066528000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767360.693973:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767360.693977:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767360.693981:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767360.693989:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767360.694008:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767360.694018:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767360/real 1775767360] req@ffff9d8188740700 x1862023066528128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767360.694032:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066528128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767360.694042:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767360.694046:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767360.694050:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767361.333065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767361.716964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767362.357031:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767362.740987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767363.381078:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767363.764991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767364.405062:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767364.788996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767365.429012:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767365.812925:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767365.812934:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767365.812937:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767365.812945:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4197 00000100:00080000:3.0:1775767365.812954:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767365.812960:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767365.812991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775767365.812994:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066528256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767365.813020:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767365.813022:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767365.813024:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767365.813026:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4197 00000100:00080000:3.0:1775767365.813029:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767365.813031:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767365.813038:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767365.813047:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066528384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767365.813054:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767365.813057:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767365.813058:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767365.813061:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4197 00000100:00080000:3.0:1775767365.813063:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767365.813065:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767365.813068:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767365.813076:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066528256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767365.813076:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066528512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767365.813091:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767365.813093:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767365.813094:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767365.813097:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4197 00000100:00080000:3.0:1775767365.813099:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767365.813101:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767365.813104:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767365.813113:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066528640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767365.813118:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767365.813120:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767365.813121:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767365.813124:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4197 00000100:00080000:3.0:1775767365.813136:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767365.813139:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767365.813142:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767365.813147:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767365.813151:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066528768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767365.813197:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767365/real 1775767365] req@ffff9d8188742a00 x1862023066528256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767365.813217:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066528256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767365.813225:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767365.813228:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767365.813231:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066528256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767365.813237:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767365.813256:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767365.813265:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767365/real 1775767365] req@ffff9d8188741c00 x1862023066528384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767365.813295:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066528384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767365.813305:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767365.813308:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767365.813311:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767365.813334:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767365.813356:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767365.813368:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767365/real 1775767365] req@ffff9d8188740380 x1862023066528512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767365.813384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066528512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767365.813390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767365.813392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767365.813394:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767365.813399:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767365.813417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767365.813429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767365/real 1775767365] req@ffff9d8188740700 x1862023066528640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767365.813442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066528640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767365.813453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767365.813456:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767365.813459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767365.813465:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767365.813481:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767365.813488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767365/real 1775767365] req@ffff9d8188740000 x1862023066528768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767365.813501:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066528768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767365.813506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767365.813508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767365.813510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066528768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767366.453012:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767366.836960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767367.477025:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767367.860993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767368.501028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767368.885004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767369.525039:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767369.908975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767370.549165:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767370.932989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767370.933002:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767370.933009:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767370.933013:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767370.933024:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4202 00000100:00080000:1.0:1775767370.933035:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767370.933043:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767370.933095:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066528896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767370.933121:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767370.933126:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767370.933128:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767370.933132:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4202 00000100:00080000:1.0:1775767370.933136:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767370.933141:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767370.933150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767370.933163:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066529024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767370.933174:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767370.933178:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767370.933180:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767370.933185:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4202 00000100:00080000:1.0:1775767370.933189:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767370.933193:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767370.933199:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767370.933202:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066528896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767370.933213:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066529152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767370.933225:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767370.933229:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767370.933231:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767370.933235:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4202 00000100:00080000:1.0:1775767370.933283:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767370.933288:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767370.933294:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767370.933295:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767370.933311:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066529280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767370.933330:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767370.933333:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767370.933335:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767370.933341:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4202 00000100:00080000:1.0:1775767370.933345:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767370.933347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767370/real 1775767370] req@ffff9d8190550e00 x1862023066528896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767370.933353:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767370.933363:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767370.933371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066528896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767370.933379:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066529408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767370.933381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767370.933390:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767370.933397:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066528896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767370.933405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767370.933424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767370.933442:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767370/real 1775767370] req@ffff9d8190551880 x1862023066529024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767370.933450:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066529024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767370.933456:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767370.933459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767370.933461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767370.933478:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767370.933488:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767370.933493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767370/real 1775767370] req@ffff9d8190551500 x1862023066529152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767370.933500:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066529152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767370.933516:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767370.933518:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767370.933520:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767370.933524:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767370.933533:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767370.933539:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767370/real 1775767370] req@ffff9d8190553100 x1862023066529280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767370.933546:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066529280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767370.933551:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767370.933553:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767370.933556:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767370.933559:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767370.933568:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767370.933573:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767370/real 1775767370] req@ffff9d8190553480 x1862023066529408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767370.933580:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066529408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767370.933585:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767370.933587:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767370.933589:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767371.573097:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767371.956973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767372.597103:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767372.981024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767373.621030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767374.004977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767374.645067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767375.028988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767375.669011:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767376.052990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767376.053076:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767376.053082:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767376.053086:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767376.053095:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4207 00000100:00080000:0.0:1775767376.053106:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767376.053112:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767376.053163:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066529536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767376.053288:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767376.053291:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767376.053293:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767376.053296:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4207 00000100:00080000:0.0:1775767376.053300:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767376.053303:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767376.053311:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767376.053340:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066529664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767376.053351:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767376.053356:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767376.053356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066529536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767376.053360:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767376.053369:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4207 00000100:00080000:0.0:1775767376.053371:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767376.053374:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767376.053378:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767376.053388:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066529792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767376.053395:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767376.053398:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767376.053399:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767376.053402:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4207 00000100:00080000:0.0:1775767376.053408:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767376.053410:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767376.053414:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767376.053419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767376.053428:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066529920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767376.053441:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767376.053444:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767376.053445:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767376.053448:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4207 00000100:00080000:0.0:1775767376.053452:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767376.053455:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767376.053459:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767376.053464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767376/real 1775767376] req@ffff9d81901afb80 x1862023066529536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767376.053481:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066530048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767376.053499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066529536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767376.053509:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767376.053513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767376.053517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066529536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767376.053565:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767376.053582:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767376.053590:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767376/real 1775767376] req@ffff9d81901aed80 x1862023066529664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767376.053600:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066529664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767376.053606:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767376.053608:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767376.053621:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767376.053625:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767376.053636:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767376.053642:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767376/real 1775767376] req@ffff9d81901af100 x1862023066529792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767376.053650:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066529792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767376.053656:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767376.053658:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767376.053660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767376.053664:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767376.053675:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767376.053680:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767376/real 1775767376] req@ffff9d81901ae680 x1862023066529920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767376.053688:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066529920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767376.053700:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767376.053703:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767376.053705:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066529920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767376.053709:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767376.053720:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767376.053726:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767376/real 1775767376] req@ffff9d81901ad180 x1862023066530048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767376.053733:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066530048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767376.053740:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767376.053742:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767376.053744:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767376.693129:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767377.076976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767377.717050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767378.100949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767378.741077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767379.125036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767379.765057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767380.149027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767380.789050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767381.172964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767381.173021:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767381.173033:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767381.173038:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767381.173053:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4212 00000100:00080000:1.0:1775767381.173067:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767381.173076:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767381.173130:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066530176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767381.173199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767381.173203:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767381.173204:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767381.173208:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4212 00000100:00080000:1.0:1775767381.173212:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767381.173215:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767381.173226:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767381.173241:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066530304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767381.173254:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767381.173257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066530176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767381.173258:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767381.173262:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767381.173272:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4212 00000100:00080000:1.0:1775767381.173276:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767381.173281:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767381.173287:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767381.173301:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066530432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767381.173317:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767381.173321:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767381.173322:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767381.173325:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4212 00000100:00080000:1.0:1775767381.173335:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767381.173338:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767381.173343:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767381.173352:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066530560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767381.173356:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767381.173364:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767381.173367:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767381.173369:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767381.173372:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4212 00000100:00080000:1.0:1775767381.173374:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767381.173377:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767381.173380:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767381.173386:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066530688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767381.173420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767381/real 1775767381] req@ffff9d8190553480 x1862023066530176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767381.173438:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066530176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767381.173452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767381.173455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767381.173459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066530176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767381.173481:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767381.173499:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767381.173508:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767381/real 1775767381] req@ffff9d8190553100 x1862023066530304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767381.173516:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066530304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767381.173522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767381.173524:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767381.173526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767381.173530:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767381.173541:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767381.173546:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767381/real 1775767381] req@ffff9d8190551500 x1862023066530432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767381.173554:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066530432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767381.173559:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767381.173562:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767381.173564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767381.173568:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767381.173580:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767381.173585:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767381/real 1775767381] req@ffff9d8190551880 x1862023066530560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767381.173600:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066530560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767381.173607:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767381.173609:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767381.173611:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767381.173616:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767381.173635:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767381.173646:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767381/real 1775767381] req@ffff9d8190550e00 x1862023066530688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767381.173658:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066530688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767381.173668:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767381.173672:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767381.173675:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767381.812992:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767382.196988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767382.837069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767383.220972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767383.861040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767384.244997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767384.885019:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767385.269021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767385.909045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767386.292961:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767386.292967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767386.292974:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767386.292979:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767386.293001:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4217 00000100:00080000:1.0:1775767386.293012:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767386.293021:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767386.293060:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066530816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767386.293089:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767386.293119:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767386.293121:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767386.293136:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4217 00000100:00080000:1.0:1775767386.293140:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767386.293143:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767386.293150:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767386.293163:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066530944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767386.293175:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767386.293177:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767386.293179:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767386.293182:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4217 00000100:00080000:1.0:1775767386.293184:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767386.293187:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767386.293201:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767386.293209:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066531072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767386.293216:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767386.293219:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767386.293220:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767386.293220:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066530816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767386.293225:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4217 00000100:00080000:1.0:1775767386.293230:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767386.293232:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767386.293236:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767386.293245:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066531200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767386.293251:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767386.293254:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767386.293255:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767386.293257:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4217 00000100:00080000:1.0:1775767386.293260:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767386.293262:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767386.293266:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767386.293278:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066531328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767386.293289:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767386.293355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767386/real 1775767386] req@ffff9d8190550e00 x1862023066530816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767386.293377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066530816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767386.293385:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767386.293388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767386.293391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066530816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767386.293397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767386.293415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767386.293423:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767386/real 1775767386] req@ffff9d8190551880 x1862023066530944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767386.293440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066530944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767386.293446:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767386.293448:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767386.293450:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066530944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767386.293466:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767386.293476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767386.293482:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767386/real 1775767386] req@ffff9d8190551500 x1862023066531072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767386.293490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066531072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767386.293506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767386.293507:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767386.293509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767386.293513:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767386.293523:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767386.293528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767386/real 1775767386] req@ffff9d8190553100 x1862023066531200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767386.293535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066531200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767386.293542:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767386.293544:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767386.293546:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767386.293552:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767386.293570:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767386.293581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767386/real 1775767386] req@ffff9d8190553480 x1862023066531328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767386.293593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066531328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767386.293609:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767386.293613:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767386.293615:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767386.933029:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767387.316977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767387.957052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767388.340982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767388.981127:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767389.364946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767390.005026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767390.389015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767391.029046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767391.412934:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767391.412990:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767391.412996:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767391.413000:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767391.413007:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4222 00000100:00080000:3.0:1775767391.413015:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767391.413020:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767391.413069:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066531456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767391.413106:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767391.413110:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767391.413112:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767391.413116:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4222 00000100:00080000:3.0:1775767391.413120:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767391.413125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767391.413133:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767391.413144:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066531584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767391.413154:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767391.413156:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066531456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767391.413158:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767391.413161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767391.413166:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4222 00000100:00080000:3.0:1775767391.413168:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767391.413170:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767391.413173:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767391.413182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066531712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767391.413188:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767391.413191:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767391.413193:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767391.413195:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4222 00000100:00080000:3.0:1775767391.413198:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767391.413200:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767391.413203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767391.413209:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066531840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767391.413215:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767391.413217:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767391.413219:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767391.413221:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4222 00000100:00080000:3.0:1775767391.413224:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767391.413226:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767391.413230:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767391.413239:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066531968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767391.413266:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767391.413292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767391.413307:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767391.413320:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767392.053109:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767392.436961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767393.077014:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767393.461005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767394.101082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767394.485068:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767394.485131:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767394.485170:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767394.485187:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767394.485231:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767394.485240:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767394.485250:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767394.485260:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767394.485279:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767391/real 1775767394] req@ffff9d8188741f80 x1862023066531584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767394.485314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066531584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767394.485328:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767394.485333:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767394.485337:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767394.485345:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767391/real 1775767394] req@ffff9d8188743800 x1862023066531712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767394.485359:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066531712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767394.485367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767394.485371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767394.485375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767394.485382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767391/real 1775767394] req@ffff9d8188741880 x1862023066531840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767394.485396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066531840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767394.485406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767394.485409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767394.485413:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767394.485420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767391/real 1775767394] req@ffff9d8188742680 x1862023066531968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767394.485433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066531968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767394.485442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767394.485445:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767394.485449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066531968:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767394.485455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767391/real 1775767394] req@ffff9d8188742300 x1862023066531456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767394.485467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066531456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767394.485475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767394.485478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767394.485481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066531456:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775767395.125036:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767395.509026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767396.149113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767396.532960:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767396.532969:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767396.532973:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767396.532981:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4228 00000100:00080000:3.0:1775767396.532996:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767396.533012:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767396.533034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775767396.533043:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066532096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767396.533079:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767396.533082:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767396.533083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767396.533086:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4228 00000100:00080000:3.0:1775767396.533088:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767396.533090:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767396.533097:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767396.533105:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066532224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767396.533112:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767396.533115:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767396.533116:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767396.533119:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4228 00000100:00080000:3.0:1775767396.533121:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767396.533123:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767396.533127:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767396.533132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066532096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767396.533136:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066532352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767396.533144:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767396.533146:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767396.533148:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767396.533151:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4228 00000100:00080000:3.0:1775767396.533153:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767396.533155:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767396.533158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767396.533167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066532480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767396.533173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767396.533176:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767396.533177:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767396.533179:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4228 00000100:00080000:3.0:1775767396.533182:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767396.533184:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767396.533187:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767396.533190:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767396.533198:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066532608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767396.533252:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767396/real 1775767396] req@ffff9d8188740e00 x1862023066532096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767396.533271:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066532096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767396.533281:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767396.533286:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767396.533291:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066532096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767396.533299:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767396.533332:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767396.533346:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767396/real 1775767396] req@ffff9d8188741180 x1862023066532224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767396.533358:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066532224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767396.533365:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767396.533369:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767396.533372:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767396.533395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767396.533421:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767396.533433:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767396/real 1775767396] req@ffff9d8188743100 x1862023066532352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767396.533444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066532352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767396.533452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767396.533455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767396.533458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767396.533463:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767396.533483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767396.533493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767396/real 1775767396] req@ffff9d8188743480 x1862023066532480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767396.533504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066532480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767396.533533:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767396.533537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767396.533540:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767396.533547:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767396.533568:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767396.533579:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767396/real 1775767396] req@ffff9d8188743b80 x1862023066532608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767396.533590:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066532608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767396.533598:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767396.533601:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767396.533604:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767397.173035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767397.557098:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767398.197114:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767398.580973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767399.221148:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767399.605059:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767400.245148:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767400.629053:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767401.269089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767401.652990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767401.653002:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767401.653008:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767401.653012:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767401.653020:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4233 00000100:00080000:0.0:1775767401.653028:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767401.653035:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767401.653068:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066532736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767401.653095:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767401.653097:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767401.653099:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767401.653102:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4233 00000100:00080000:0.0:1775767401.653104:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767401.653106:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767401.653114:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767401.653122:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066532864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767401.653129:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767401.653132:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767401.653133:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767401.653136:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4233 00000100:00080000:0.0:1775767401.653138:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767401.653141:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767401.653145:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767401.653150:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066532736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767401.653155:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066532992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767401.653163:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767401.653165:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767401.653166:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767401.653169:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4233 00000100:00080000:0.0:1775767401.653172:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767401.653173:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767401.653177:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767401.653186:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066533120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767401.653191:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767401.653193:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767401.653194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767401.653197:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4233 00000100:00080000:0.0:1775767401.653200:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767401.653202:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767401.653205:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775767401.653209:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767401.653215:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066533248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767401.653251:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767401/real 1775767401] req@ffff9d81901ad180 x1862023066532736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767401.653264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066532736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767401.653272:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767401.653275:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767401.653278:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066532736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767401.653283:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767401.653298:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767401.653306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767401/real 1775767401] req@ffff9d81901ae680 x1862023066532864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767401.653313:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066532864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767401.653323:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767401.653325:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767401.653328:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767401.653343:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767401.653354:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767401.653359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767401/real 1775767401] req@ffff9d81901af100 x1862023066532992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767401.653366:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066532992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767401.653372:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767401.653373:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767401.653375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066532992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767401.653379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767401.653389:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767401.653394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767401/real 1775767401] req@ffff9d81901aed80 x1862023066533120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767401.653401:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066533120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767401.653407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767401.653409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767401.653411:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767401.653415:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767401.653424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767401.653429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767401/real 1775767401] req@ffff9d81901afb80 x1862023066533248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767401.653436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066533248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767401.653441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767401.653443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767401.653445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767402.293128:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767402.677034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767403.317186:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767403.701035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767404.341024:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767404.724969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767405.365018:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767405.748991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767406.389057:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767406.773017:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767406.773020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767406.773027:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767406.773031:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767406.773039:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4238 00000100:00080000:0.0:1775767406.773047:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767406.773053:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767406.773087:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066533376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767406.773109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767406.773112:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767406.773113:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767406.773116:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4238 00000100:00080000:0.0:1775767406.773118:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767406.773120:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767406.773128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767406.773136:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066533504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767406.773144:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767406.773147:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767406.773149:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767406.773149:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066533376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767406.773157:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4238 00000100:00080000:0.0:1775767406.773161:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767406.773163:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767406.773166:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767406.773174:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066533632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767406.773180:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767406.773182:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767406.773184:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767406.773186:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4238 00000100:00080000:0.0:1775767406.773189:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767406.773191:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767406.773195:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767406.773201:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066533760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767406.773206:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767406.773208:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767406.773210:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767406.773212:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4238 00000100:00080000:0.0:1775767406.773215:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767406.773219:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775767406.773221:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767406.773225:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767406.773232:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066533888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767406.773260:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767406/real 1775767406] req@ffff9d81901afb80 x1862023066533376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767406.773275:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066533376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767406.773282:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767406.773285:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767406.773288:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066533376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767406.773306:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767406.773323:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767406.773331:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767406/real 1775767406] req@ffff9d81901aed80 x1862023066533504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767406.773339:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066533504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767406.773344:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767406.773346:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767406.773348:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767406.773352:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767406.773362:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767406.773367:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767406/real 1775767406] req@ffff9d81901af100 x1862023066533632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767406.773374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066533632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767406.773379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767406.773381:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767406.773383:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767406.773386:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767406.773397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767406.773403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767406/real 1775767406] req@ffff9d81901ae680 x1862023066533760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767406.773410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066533760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767406.773416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767406.773418:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767406.773420:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767406.773423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767406.773433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767406.773439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767406/real 1775767406] req@ffff9d81901ad180 x1862023066533888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767406.773446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066533888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767406.773451:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767406.773453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767406.773455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066533888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767407.413014:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767407.797004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767408.437079:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767408.820973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767409.461029:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767409.844973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767410.485059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767410.868949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767411.509059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767411.892970:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767411.892974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767411.892980:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767411.892984:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767411.892992:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4243 00000100:00080000:0.0:1775767411.893001:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767411.893007:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767411.893041:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066534016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767411.893069:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767411.893071:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767411.893073:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767411.893075:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4243 00000100:00080000:0.0:1775767411.893077:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767411.893080:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767411.893087:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767411.893095:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066534144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767411.893103:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767411.893105:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767411.893107:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767411.893110:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4243 00000100:00080000:0.0:1775767411.893113:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767411.893115:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066534016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767411.893116:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767411.893122:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767411.893129:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066534272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767411.893135:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767411.893137:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767411.893139:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767411.893141:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4243 00000100:00080000:0.0:1775767411.893144:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767411.893146:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767411.893150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767411.893157:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066534400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767411.893162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767411.893164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767411.893165:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767411.893167:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4243 00000100:00080000:0.0:1775767411.893170:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767411.893173:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767411.893175:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775767411.893182:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767411.893184:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066534528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767411.893229:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767411/real 1775767411] req@ffff9d81901ad180 x1862023066534016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767411.893256:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066534016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767411.893263:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767411.893266:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767411.893269:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066534016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767411.893290:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767411.893325:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767411.893339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767411/real 1775767411] req@ffff9d81901ae680 x1862023066534144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767411.893352:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066534144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767411.893361:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767411.893364:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767411.893366:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767411.893371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767411.893387:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767411.893394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767411/real 1775767411] req@ffff9d81901af100 x1862023066534272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767411.893402:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066534272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767411.893411:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767411.893414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767411.893416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767411.893419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767411.893430:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767411.893435:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767411/real 1775767411] req@ffff9d81901aed80 x1862023066534400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767411.893442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066534400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767411.893448:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767411.893450:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767411.893452:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767411.893455:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767411.893466:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767411.893471:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767411/real 1775767411] req@ffff9d81901afb80 x1862023066534528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767411.893478:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066534528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767411.893483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767411.893485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767411.893487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767412.533064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767412.916968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767413.557099:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767413.940981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767414.580996:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767414.964945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767415.605049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767415.988966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767416.629058:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767417.012997:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767417.013005:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767417.013009:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767417.013011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767417.013019:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4248 00000100:00080000:3.0:1775767417.013032:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767417.013039:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767417.013075:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066534656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767417.013100:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767417.013103:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767417.013104:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767417.013107:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4248 00000100:00080000:3.0:1775767417.013109:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767417.013112:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767417.013120:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767417.013130:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066534784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767417.013135:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066534656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767417.013142:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767417.013146:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767417.013147:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767417.013151:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4248 00000100:00080000:3.0:1775767417.013153:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767417.013155:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767417.013159:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767417.013168:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066534912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767417.013177:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767417.013181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767417.013183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767417.013186:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4248 00000100:00080000:3.0:1775767417.013199:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767417.013203:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767417.013208:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767417.013210:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767417.013228:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066535040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767417.013249:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767417.013250:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767417/real 1775767417] req@ffff9d8188741500 x1862023066534656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767417.013260:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767417.013265:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767417.013273:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4248 00000100:00100000:0.0:1775767417.013277:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066534656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767417.013279:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767417.013288:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775767417.013290:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775767417.013294:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767417.013297:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767417.013302:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066534656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767417.013311:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066535168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767417.013324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767417.013343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767417.013351:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767417/real 1775767417] req@ffff9d8188740a80 x1862023066534784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767417.013359:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066534784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767417.013365:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767417.013367:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767417.013370:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767417.013374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767417.013385:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767417.013390:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767417/real 1775767417] req@ffff9d8188742d80 x1862023066534912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767417.013397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066534912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767417.013403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767417.013404:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767417.013406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066534912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767417.013410:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767417.013421:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767417.013426:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767417/real 1775767417] req@ffff9d8188743b80 x1862023066535040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767417.013433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066535040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767417.013439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767417.013441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767417.013443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767417.013446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767417.013455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767417.013460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767417/real 1775767417] req@ffff9d8188743480 x1862023066535168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767417.013467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066535168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767417.013473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767417.013475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767417.013476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767417.653065:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767418.036972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767418.677052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767419.060988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767419.701066:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767420.084997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767420.725066:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767421.109003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767421.749082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767422.132966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767422.133003:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767422.133010:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767422.133015:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767422.133025:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4253 00000100:00080000:1.0:1775767422.133035:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767422.133043:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767422.133081:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066535296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767422.133108:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767422.133111:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767422.133113:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767422.133117:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4253 00000100:00080000:1.0:1775767422.133121:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767422.133125:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767422.133134:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767422.133149:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066535424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767422.133162:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066535296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767422.133162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767422.133169:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767422.133173:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767422.133179:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4253 00000100:00080000:1.0:1775767422.133182:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767422.133186:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767422.133191:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767422.133205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066535552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767422.133216:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767422.133219:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767422.133221:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767422.133227:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4253 00000800:00000100:0.0:1775767422.133229:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767422.133233:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767422.133238:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767422.133244:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767422.133256:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066535680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767422.133273:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767422.133276:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767422.133278:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767422.133282:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4253 00000100:00080000:1.0:1775767422.133286:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767422.133289:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767422/real 1775767422] req@ffff9d8190553480 x1862023066535296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767422.133293:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767422.133302:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767422.133319:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066535808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767422.133329:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066535296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767422.133342:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767422.133348:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767422.133352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066535296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767422.133374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767422.133398:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767422.133412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767422/real 1775767422] req@ffff9d8190553100 x1862023066535424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767422.133425:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066535424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767422.133434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767422.133438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767422.133441:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767422.133447:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767422.133466:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767422.133476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767422/real 1775767422] req@ffff9d8190551500 x1862023066535552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767422.133489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066535552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767422.133505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767422.133508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767422.133511:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767422.133517:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767422.133536:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767422.133546:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767422/real 1775767422] req@ffff9d8190551880 x1862023066535680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767422.133558:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066535680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767422.133567:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767422.133570:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767422.133573:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767422.133579:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767422.133598:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767422.133609:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767422/real 1775767422] req@ffff9d8190550e00 x1862023066535808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767422.133621:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066535808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767422.133630:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767422.133633:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767422.133637:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066535808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767422.773039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767423.157016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767423.797036:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767424.180981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767424.821074:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767425.204985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767425.845059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767426.228964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767426.869021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767427.253007:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767427.253008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767427.253019:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767427.253036:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767427.253051:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4258 00000100:00080000:1.0:1775767427.253063:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767427.253072:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767427.253122:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066535936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767427.253150:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767427.253153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767427.253154:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767427.253157:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4258 00000100:00080000:1.0:1775767427.253160:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767427.253162:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767427.253173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767427.253181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066536064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767427.253189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767427.253192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767427.253193:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767427.253196:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4258 00000100:00080000:1.0:1775767427.253198:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767427.253200:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767427.253203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767427.253210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066536192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767427.253216:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767427.253218:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767427.253219:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767427.253222:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4258 00000100:00080000:1.0:1775767427.253224:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767427.253227:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767427.253238:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767427.253244:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066535936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767427.253252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066536320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767427.253261:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767427.253264:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767427.253266:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767427.253273:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4258 00000100:00080000:1.0:1775767427.253276:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767427.253278:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767427.253282:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767427.253290:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066536448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767427.253322:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767427.253378:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767427/real 1775767427] req@ffff9d8190550e00 x1862023066535936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767427.253400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066535936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767427.253412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767427.253417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767427.253421:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066535936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767427.253428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767427.253447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767427.253457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767427/real 1775767427] req@ffff9d8190551880 x1862023066536064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767427.253465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066536064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767427.253471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767427.253474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767427.253476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767427.253480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767427.253492:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767427.253502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767427/real 1775767427] req@ffff9d8190551500 x1862023066536192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767427.253514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066536192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767427.253523:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767427.253526:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767427.253530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767427.253554:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767427.253577:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767427.253588:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767427/real 1775767427] req@ffff9d8190553100 x1862023066536320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767427.253601:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066536320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767427.253611:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767427.253615:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767427.253618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767427.253625:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767427.253644:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767427.253655:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767427/real 1775767427] req@ffff9d8190553480 x1862023066536448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767427.253675:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066536448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767427.253684:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767427.253688:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767427.253691:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767427.893096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767428.277054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767428.917057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767429.301034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767429.941046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767430.325000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767430.965044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767431.348988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767431.989032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767432.372941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767432.372955:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767432.372963:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767432.372967:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767432.372976:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4263 00000100:00080000:1.0:1775767432.372986:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767432.372994:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767432.373033:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066536576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767432.373060:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767432.373063:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767432.373065:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767432.373072:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4263 00000100:00080000:1.0:1775767432.373075:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767432.373078:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767432.373086:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767432.373097:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066536704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767432.373107:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066536576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767432.373124:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767432.373126:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767432.373127:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767432.373131:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4263 00000100:00080000:1.0:1775767432.373134:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767432.373137:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767432.373140:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767432.373148:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066536832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767432.373154:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767432.373157:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767432.373158:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767432.373160:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4263 00000100:00080000:1.0:1775767432.373163:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767432.373166:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767432.373169:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767432.373173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767432.373180:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066536960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767432.373187:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767432.373189:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767432.373191:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767432.373193:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4263 00000100:00080000:1.0:1775767432.373196:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767432.373198:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767432.373201:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767432.373209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066537088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767432.373217:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767432/real 1775767432] req@ffff9d8190553480 x1862023066536576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767432.373233:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066536576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767432.373241:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767432.373244:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767432.373246:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066536576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767432.373263:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767432.373281:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767432.373290:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767432/real 1775767432] req@ffff9d8190553100 x1862023066536704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767432.373302:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066536704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767432.373311:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767432.373315:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767432.373318:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767432.373325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767432.373345:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767432.373356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767432/real 1775767432] req@ffff9d8190551500 x1862023066536832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767432.373365:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066536832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767432.373371:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767432.373373:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767432.373375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767432.373379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767432.373394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767432.373407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767432/real 1775767432] req@ffff9d8190551880 x1862023066536960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767432.373414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066536960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767432.373420:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767432.373422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767432.373425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066536960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767432.373428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767432.373439:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767432.373444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767432/real 1775767432] req@ffff9d8190550e00 x1862023066537088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767432.373451:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066537088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767432.373457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767432.373458:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767432.373460:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767433.013033:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767433.397008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767434.037028:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767434.420959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767435.061045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767435.444990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767436.085032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767436.468971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767437.109076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767437.493010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767437.493019:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767437.493027:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767437.493031:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767437.493040:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4269 00000100:00080000:3.0:1775767437.493052:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767437.493060:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767437.493099:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066537216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767437.493139:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767437.493143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767437.493149:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767437.493153:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4269 00000100:00080000:3.0:1775767437.493157:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767437.493162:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767437.493172:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767437.493188:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066537344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767437.493198:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066537216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767437.493204:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767437.493209:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767437.493211:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767437.493216:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4269 00000100:00080000:3.0:1775767437.493221:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767437.493225:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767437.493230:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767437.493244:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066537472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767437.493255:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767437.493256:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767437.493262:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767437.493264:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767437.493271:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4269 00000100:00080000:3.0:1775767437.493276:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767437.493281:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767437.493288:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767437.493298:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767437/real 1775767437] req@ffff9d8188743100 x1862023066537216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767437.493304:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066537600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767437.493318:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767437.493319:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066537216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767437.493329:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767437.493334:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767437.493336:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767437.493341:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4269 00000100:00080000:0.0:1775767437.493343:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767437.493349:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066537216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767437.493349:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767437.493358:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767437.493364:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767437.493372:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767437.493380:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066537728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767437.493395:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767437.493406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767437/real 1775767437] req@ffff9d8188741180 x1862023066537344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.493417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066537344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.493422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.493425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767437.493427:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767437.493430:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767437.493440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767437.493446:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767437/real 1775767437] req@ffff9d8188740e00 x1862023066537472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.493453:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066537472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.493458:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.493460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767437.493462:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767437.493465:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767437.493475:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767437.493480:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767437/real 1775767437] req@ffff9d8188742300 x1862023066537600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.493492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066537600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.493498:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.493500:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767437.493502:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767437.493510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767437.493520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767437.493525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767437/real 1775767437] req@ffff9d8188742680 x1862023066537728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767437.493533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066537728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767437.493538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767437.493539:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767437.493541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767438.133043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767438.516983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767439.157058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767439.540982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767440.181037:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767440.564990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767441.205079:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767441.588952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767442.229043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767442.612979:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767442.612987:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1775767442.612987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767442.612991:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767442.612999:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4274 00000100:00080000:2.0:1775767442.613009:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767442.613016:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767442.613049:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066537856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767442.613072:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767442.613075:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767442.613076:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767442.613078:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4274 00000100:00080000:2.0:1775767442.613081:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767442.613083:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767442.613089:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767442.613096:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066537984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767442.613103:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767442.613105:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767442.613106:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767442.613108:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4274 00000100:00080000:2.0:1775767442.613110:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767442.613112:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767442.613114:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767442.613119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066538112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767442.613125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767442.613127:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767442.613129:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767442.613132:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4274 00000100:00080000:2.0:1775767442.613135:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767442.613137:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767442.613140:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767442.613146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066538240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767442.613152:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767442.613154:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767442.613157:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767442.613158:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066537856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767442.613162:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4274 00000100:00080000:2.0:1775767442.613167:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767442.613170:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767442.613173:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767442.613180:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066538368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767442.613232:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767442.613278:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767442/real 1775767442] req@ffff9d81984a0e00 x1862023066537856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767442.613298:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066537856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767442.613309:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767442.613313:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767442.613318:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066537856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767442.613326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767442.613350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767442.613360:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767442/real 1775767442] req@ffff9d81984a3480 x1862023066537984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767442.613369:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066537984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767442.613375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767442.613377:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767442.613379:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066537984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767442.613383:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767442.613395:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767442.613400:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767442/real 1775767442] req@ffff9d81984a2d80 x1862023066538112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767442.613407:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066538112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767442.613412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767442.613414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767442.613416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767442.613436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767442.613455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767442.613466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767442/real 1775767442] req@ffff9d81984a0700 x1862023066538240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767442.613480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066538240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767442.613490:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767442.613493:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767442.613496:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767442.613502:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767442.613518:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767442.613525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767442/real 1775767442] req@ffff9d81984a2a00 x1862023066538368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767442.613532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066538368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767442.613538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767442.613540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767442.613542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767443.253075:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767443.636998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767444.277082:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767444.660979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767445.301062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767445.684982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767446.325144:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767446.709000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767447.349097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767447.732998:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767447.733005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767447.733009:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767447.733018:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767447.733031:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4279 00000100:00080000:3.0:1775767447.733044:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767447.733051:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767447.733100:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066538496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767447.733144:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767447.733147:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767447.733148:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767447.733152:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4279 00000100:00080000:3.0:1775767447.733155:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767447.733159:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767447.733170:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767447.733182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066538624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767447.733191:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767447.733194:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767447.733201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767447.733202:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066538496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767447.733211:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4279 00000100:00080000:3.0:1775767447.733216:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767447.733219:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767447.733222:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767447.733235:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066538752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767447.733243:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767447.733246:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767447.733247:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767447.733250:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4279 00000100:00080000:3.0:1775767447.733261:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767447.733266:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767447.733271:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767447.733277:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767447.733297:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066538880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767447.733309:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767447.733313:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767447.733315:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767447.733320:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4279 00000100:00100000:0.0:1775767447.733326:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767447/real 1775767447] req@ffff9d8188741880 x1862023066538496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767447.733328:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767447.733343:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767447.733353:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066538496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767447.733354:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767447.733376:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767447.733377:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066539008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767447.733392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767447.733405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066538496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767447.733438:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767447.733469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767447.733479:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767447/real 1775767447] req@ffff9d8188743800 x1862023066538624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767447.733489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066538624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767447.733495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767447.733497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767447.733500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767447.733505:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767447.733544:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767447.733554:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767447/real 1775767447] req@ffff9d8188741f80 x1862023066538752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767447.733574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066538752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767447.733583:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767447.733587:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767447.733590:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767447.733598:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767447.733621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767447.733633:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767447/real 1775767447] req@ffff9d8188740000 x1862023066538880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767447.733646:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066538880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767447.733656:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767447.733659:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767447.733661:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066538880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767447.733666:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767447.733682:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767447.733689:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767447/real 1775767447] req@ffff9d8188740700 x1862023066539008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767447.733697:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066539008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767447.733706:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767447.733709:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767447.733719:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767448.373086:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767448.756985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767449.397027:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767449.780972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767450.421065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767450.805012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767451.445057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767451.828986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767452.469042:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767452.852977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767452.852999:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767452.853005:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767452.853009:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767452.853017:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4284 00000100:00080000:0.0:1775767452.853026:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767452.853031:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767452.853065:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066539136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767452.853088:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767452.853091:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767452.853092:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767452.853095:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4284 00000100:00080000:0.0:1775767452.853098:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767452.853100:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767452.853108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767452.853116:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066539264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767452.853125:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066539136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767452.853126:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767452.853134:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767452.853137:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767452.853141:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4284 00000100:00080000:0.0:1775767452.853143:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767452.853145:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767452.853148:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767452.853156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066539392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767452.853162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767452.853164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767452.853165:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767452.853168:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4284 00000100:00080000:0.0:1775767452.853184:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767452.853187:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767452.853190:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767452.853195:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767452.853201:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066539520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767452.853209:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767452.853211:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767452.853213:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767452.853215:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4284 00000100:00080000:0.0:1775767452.853217:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767452.853219:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767452.853223:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767452.853231:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066539648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767452.853241:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767452/real 1775767452] req@ffff9d81901afb80 x1862023066539136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767452.853258:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066539136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767452.853271:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767452.853274:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767452.853277:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066539136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767452.853294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767452.853310:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767452.853319:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767452/real 1775767452] req@ffff9d81901aed80 x1862023066539264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767452.853327:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066539264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767452.853333:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767452.853335:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767452.853337:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767452.853358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767452.853376:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767452.853387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767452/real 1775767452] req@ffff9d81901af100 x1862023066539392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767452.853400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066539392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767452.853424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767452.853428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767452.853431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767452.853437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767452.853453:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767452.853463:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767452/real 1775767452] req@ffff9d81901ae680 x1862023066539520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767452.853512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066539520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767452.853523:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767452.853526:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767452.853529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767452.853533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767452.853550:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767452.853557:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767452/real 1775767452] req@ffff9d81901ad180 x1862023066539648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767452.853565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066539648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767452.853571:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767452.853573:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767452.853575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767453.493031:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767453.876939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767454.517035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767454.901088:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767455.541031:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767455.925124:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767456.565065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767456.948991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767457.589060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767457.973016:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767457.973022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767457.973032:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767457.973038:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767457.973051:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4289 00000100:00080000:0.0:1775767457.973063:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767457.973071:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767457.973121:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066539776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767457.973148:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767457.973151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767457.973153:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767457.973156:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4289 00000100:00080000:0.0:1775767457.973158:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767457.973161:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767457.973170:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767457.973178:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066539904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767457.973186:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767457.973188:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767457.973190:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767457.973194:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4289 00000100:00080000:0.0:1775767457.973197:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767457.973200:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767457.973203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767457.973210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066540032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767457.973218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767457.973221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767457.973223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767457.973226:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4289 00000100:00080000:0.0:1775767457.973229:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767457.973233:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767457.973241:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767457.973242:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066539776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767457.973251:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066540160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767457.973266:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767457.973269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767457.973270:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767457.973274:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4289 00000100:00080000:0.0:1775767457.973275:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767457.973278:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767457.973281:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767457.973289:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066540288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767457.973381:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767457.973434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767457.973474:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767457.973540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767458.613078:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767458.996960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767459.637058:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767460.020983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767460.661122:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767461.044956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767461.045151:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767461.045193:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767461.045213:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767461.045250:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767461.045258:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767461.045287:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775767461.045293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767457/real 1775767461] req@ffff9d81901ad180 x1862023066539776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1775767461.045300:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775767461.045326:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066539776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767461.045336:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767461.045342:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767461.045346:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066539776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767461.045354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767457/real 1775767461] req@ffff9d81901ae680 x1862023066539904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767461.045372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066539904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767461.045380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767461.045384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767461.045401:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066539904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767461.045408:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767457/real 1775767461] req@ffff9d81901af100 x1862023066540032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767461.045420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066540032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767461.045426:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767461.045429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767461.045432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767461.045455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767457/real 1775767461] req@ffff9d81901aed80 x1862023066540160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767461.045466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066540160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767461.045475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767461.045478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767461.045481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767461.045488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767457/real 1775767461] req@ffff9d81901afb80 x1862023066540288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767461.045500:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066540288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767461.045518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767461.045522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767461.045526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767461.685060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767462.068979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767462.709090:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767463.093041:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767463.093053:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767463.093059:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767463.093073:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4294 00000100:00100000:2.0:1775767463.093077:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767463.093088:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767463.093099:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767463.093152:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066540416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767463.093178:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767463.093181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767463.093183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767463.093186:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4294 00000100:00080000:3.0:1775767463.093189:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767463.093192:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767463.093203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767463.093213:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066540544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767463.093217:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066540416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767463.093229:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767463.093232:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767463.093233:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767463.093237:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4294 00000100:00080000:3.0:1775767463.093239:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767463.093243:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767463.093247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767463.093255:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066540672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767463.093262:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767463.093266:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767463.093267:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767463.093269:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4294 00000100:00080000:3.0:1775767463.093293:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767463.093297:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767463.093303:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767463.093309:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767463.093327:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066540800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767463.093339:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767463.093343:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767463.093345:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767463.093350:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4294 00000100:00080000:3.0:1775767463.093356:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767463.093362:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767463.093364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767463/real 1775767463] req@ffff9d8188740380 x1862023066540416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767463.093388:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767463.093402:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066540416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767463.093405:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066540928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767463.093429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767463.093438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767463.093445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066540416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767463.093476:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767463.093503:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767463.093518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767463/real 1775767463] req@ffff9d8188741c00 x1862023066540544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767463.093532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066540544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767463.093542:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767463.093546:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767463.093549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767463.093559:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767463.093583:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767463.093594:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767463/real 1775767463] req@ffff9d8188742a00 x1862023066540672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767463.093607:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066540672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767463.093617:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767463.093620:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767463.093624:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767463.093631:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767463.093652:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767463.093663:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767463/real 1775767463] req@ffff9d8188740700 x1862023066540800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767463.093677:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066540800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767463.093704:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767463.093709:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767463.093713:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767463.093719:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767463.093742:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767463.093754:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767463/real 1775767463] req@ffff9d8188740000 x1862023066540928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767463.093767:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066540928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767463.093777:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767463.093780:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767463.093783:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066540928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767463.733035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767464.116995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767464.757055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767465.140984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767465.780998:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767466.164965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767466.805027:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767467.188969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767467.829159:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767468.213000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767468.213018:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767468.213025:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767468.213029:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767468.213037:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4299 00000100:00080000:3.0:1775767468.213046:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767468.213051:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767468.213083:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066541056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767468.213111:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767468.213120:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767468.213121:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767468.213125:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4299 00000100:00080000:3.0:1775767468.213129:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767468.213134:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767468.213143:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767468.213159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066541184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767468.213167:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066541056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767468.213172:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767468.213175:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767468.213176:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767468.213180:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4299 00000100:00080000:3.0:1775767468.213183:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767468.213186:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767468.213191:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767468.213202:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066541312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767468.213209:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767468.213211:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767468.213212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767468.213214:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4299 00000100:00080000:3.0:1775767468.213226:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767468.213229:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767468.213232:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767468.213238:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767468.213242:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066541440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767468.213249:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767468.213252:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767468.213254:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767468.213256:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4299 00000100:00080000:3.0:1775767468.213258:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767468.213260:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767468.213263:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767468.213271:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066541568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767468.213283:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767468/real 1775767468] req@ffff9d8188741f80 x1862023066541056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767468.213297:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066541056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767468.213305:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767468.213314:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767468.213331:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066541056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767468.213350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767468.213367:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767468.213375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767468/real 1775767468] req@ffff9d8188743800 x1862023066541184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767468.213383:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066541184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767468.213393:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767468.213396:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767468.213398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767468.213401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767468.213411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767468.213417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767468/real 1775767468] req@ffff9d8188741880 x1862023066541312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767468.213424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066541312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767468.213429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767468.213431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767468.213432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767468.213436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767468.213446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767468.213451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767468/real 1775767468] req@ffff9d8188742680 x1862023066541440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767468.213458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066541440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767468.213465:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767468.213466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767468.213469:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767468.213479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767468.213496:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767468.213507:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767468/real 1775767468] req@ffff9d8188742300 x1862023066541568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767468.213519:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066541568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767468.213528:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767468.213532:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767468.213535:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767468.853107:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767469.236988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767469.877077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767470.260981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767470.901063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767471.284946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767471.925069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767472.308992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767472.949093:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767473.332947:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767473.332958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767473.332958:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767473.332963:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767473.332974:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4304 00000100:00080000:3.0:1775767473.332984:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767473.332992:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767473.333027:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066541696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767473.333050:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767473.333052:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767473.333053:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767473.333056:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4304 00000100:00080000:3.0:1775767473.333059:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767473.333062:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767473.333070:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767473.333079:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066541824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767473.333088:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066541696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767473.333089:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767473.333096:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767473.333099:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767473.333103:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4304 00000100:00080000:3.0:1775767473.333105:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767473.333107:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767473.333111:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767473.333119:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066541952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767473.333125:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767473.333127:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767473.333129:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767473.333132:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4304 00000100:00080000:3.0:1775767473.333145:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767473.333148:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767473.333151:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767473.333157:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767473.333160:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066542080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767473.333167:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767473.333170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767473.333172:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767473.333175:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4304 00000100:00080000:3.0:1775767473.333179:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767473.333181:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767473.333184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767473.333191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066542208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767473.333214:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767473/real 1775767473] req@ffff9d8188740e00 x1862023066541696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767473.333233:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066541696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767473.333244:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767473.333249:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767473.333252:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066541696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767473.333288:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767473.333314:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767473.333324:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767473/real 1775767473] req@ffff9d8188741180 x1862023066541824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767473.333333:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066541824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767473.333338:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767473.333340:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767473.333343:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767473.333347:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767473.333358:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767473.333364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767473/real 1775767473] req@ffff9d8188743100 x1862023066541952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767473.333371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066541952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767473.333376:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767473.333378:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767473.333380:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066541952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767473.333389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767473.333401:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767473.333407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767473/real 1775767473] req@ffff9d8188743480 x1862023066542080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767473.333420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066542080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767473.333426:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767473.333428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767473.333430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767473.333433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767473.333443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767473.333448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767473/real 1775767473] req@ffff9d8188743b80 x1862023066542208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767473.333455:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066542208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767473.333460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767473.333462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767473.333464:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767473.973061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767474.356979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767474.997032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767475.380941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767476.021044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767476.404968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767477.045086:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767477.428980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767478.069034:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767478.452965:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767478.452973:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1775767478.452975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767478.452977:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767478.452986:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4309 00000100:00080000:2.0:1775767478.452995:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767478.453002:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767478.453034:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066542336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767478.453056:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767478.453061:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767478.453062:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767478.453065:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4309 00000100:00080000:2.0:1775767478.453067:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767478.453075:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767478.453082:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767478.453095:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066542464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767478.453103:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767478.453107:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767478.453109:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767478.453114:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4309 00000100:00080000:2.0:1775767478.453117:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767478.453122:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066542336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767478.453124:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767478.453134:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767478.453149:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066542592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767478.453161:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767478.453165:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767478.453167:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767478.453170:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4309 00000100:00080000:2.0:1775767478.453178:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767478.453182:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767478.453188:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767478.453191:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767478.453204:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066542720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767478.453212:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767478.453214:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767478.453215:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767478.453233:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4309 00000100:00080000:2.0:1775767478.453237:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767478.453242:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767478.453247:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767478/real 1775767478] req@ffff9d81984a2300 x1862023066542336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775767478.453248:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767478.453266:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066542848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767478.453269:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066542336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767478.453298:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767478.453303:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767478.453307:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066542336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767478.453313:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767478.453332:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767478.453344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767478/real 1775767478] req@ffff9d81984a2a00 x1862023066542464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767478.453358:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066542464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767478.453367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767478.453371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767478.453374:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767478.453396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767478.453418:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767478.453429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767478/real 1775767478] req@ffff9d81984a0700 x1862023066542592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767478.453443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066542592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767478.453452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767478.453455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767478.453459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767478.453465:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767478.453479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767478.453486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767478/real 1775767478] req@ffff9d81984a2d80 x1862023066542720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767478.453494:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066542720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767478.453501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767478.453503:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767478.453505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767478.453508:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767478.453517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767478.453522:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767478/real 1775767478] req@ffff9d81984a3480 x1862023066542848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767478.453529:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066542848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767478.453535:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767478.453536:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767478.453538:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066542848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767479.093050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767479.477010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767480.117045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767480.500954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767481.141059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767481.525002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767482.165046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767482.548966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767483.189053:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767483.572980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767483.573000:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767483.573005:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767483.573009:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767483.573016:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4315 00000100:00080000:3.0:1775767483.573025:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767483.573030:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767483.573068:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066542976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767483.573095:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767483.573099:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767483.573101:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767483.573105:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4315 00000100:00080000:3.0:1775767483.573111:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767483.573116:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767483.573126:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767483.573129:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066542976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767483.573142:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066543104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767483.573154:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767483.573157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767483.573159:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767483.573164:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4315 00000100:00080000:3.0:1775767483.573204:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767483.573211:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767483.573212:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767483.573220:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767483.573235:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066543232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767483.573246:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767483.573250:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767483.573252:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767483.573263:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767483/real 1775767483] req@ffff9d8188742d80 x1862023066542976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767483.573266:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4315 00000100:00080000:3.0:1775767483.573275:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767483.573281:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767483.573285:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066542976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767483.573287:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775767483.573300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767483.573304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767483.573309:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066542976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767483.573309:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066543360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767483.573323:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767483.573326:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767483.573328:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767483.573334:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4315 00000100:00100000:0.0:1775767483.573335:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543104:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775767483.573340:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767483.573346:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767483.573351:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767483.573368:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066543488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767483.573369:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767483.573387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767483/real 1775767483] req@ffff9d8188740a80 x1862023066543104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767483.573404:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066543104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767483.573413:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767483.573417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767483.573420:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767483.573427:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767483.573449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767483.573460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767483/real 1775767483] req@ffff9d8188741500 x1862023066543232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767483.573469:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066543232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767483.573474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767483.573476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767483.573478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767483.573482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767483.573495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767483.573501:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767483/real 1775767483] req@ffff9d8188743b80 x1862023066543360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767483.573508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066543360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767483.573515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767483.573517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767483.573519:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767483.573529:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767483.573539:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767483.573545:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767483/real 1775767483] req@ffff9d8188743480 x1862023066543488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767483.573556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066543488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767483.573562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767483.573563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767483.573565:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767484.213061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767484.596982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767485.237049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767485.620977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767486.261073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767486.645004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767487.285108:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767487.669028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767488.309072:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767488.692979:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767488.692987:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767488.692991:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767488.693000:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4320 00000100:00080000:3.0:1775767488.693010:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767488.693013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767488.693017:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767488.693057:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066543616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767488.693088:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767488.693092:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767488.693094:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767488.693099:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4320 00000100:00080000:3.0:1775767488.693112:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767488.693142:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066543616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767488.693151:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767488.693161:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767488.693175:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066543744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767488.693185:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767488.693188:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767488.693190:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767488.693194:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4320 00000100:00080000:3.0:1775767488.693211:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767488.693214:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767488.693219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767488.693226:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767488.693233:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066543872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767488.693243:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767488.693247:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767488.693250:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767488.693257:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4320 00000100:00080000:3.0:1775767488.693261:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767488.693267:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767488.693273:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767488.693287:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767488/real 1775767488] req@ffff9d8188743100 x1862023066543616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767488.693292:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066544000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767488.693319:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767488.693323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066543616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767488.693335:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767488.693343:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767488.693347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767488.693353:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4320 00000100:00080000:0.0:1775767488.693355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775767488.693364:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767488.693365:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066543616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767488.693377:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767488.693383:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767488.693402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767488.693402:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066544128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767488.693454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767488.693472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767488/real 1775767488] req@ffff9d8188741180 x1862023066543744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767488.693482:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066543744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767488.693488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767488.693491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767488.693493:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767488.693499:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767488.693515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767488.693522:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767488/real 1775767488] req@ffff9d8188740e00 x1862023066543872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767488.693530:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066543872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767488.693535:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767488.693537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767488.693539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066543872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767488.693543:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767488.693557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767488.693566:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767488/real 1775767488] req@ffff9d8188742300 x1862023066544000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767488.693585:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066544000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767488.693596:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767488.693600:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767488.693603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767488.693621:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767488.693638:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767488.693646:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767488/real 1775767488] req@ffff9d8188742680 x1862023066544128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767488.693655:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066544128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767488.693661:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767488.693664:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767488.693666:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767489.333063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767489.717022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767490.357032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767490.740987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767491.381039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767491.764973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767492.405008:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767492.788952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767493.429067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767493.812944:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767493.812950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767493.812953:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767493.812958:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767493.812966:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4325 00000100:00080000:3.0:1775767493.812975:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767493.812980:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767493.813013:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066544256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767493.813037:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767493.813041:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767493.813043:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767493.813048:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4325 00000100:00080000:3.0:1775767493.813052:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767493.813056:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767493.813066:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066544256:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775767493.813066:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767493.813110:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066544384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767493.813118:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767493.813121:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767493.813123:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767493.813126:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4325 00000100:00080000:3.0:1775767493.813130:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767493.813132:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767493.813136:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767493.813137:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767493.813149:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066544512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767493.813156:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767493.813158:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767493.813160:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767493.813162:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4325 00000100:00080000:3.0:1775767493.813165:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767493.813167:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767493.813171:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767493.813179:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767493/real 1775767493] req@ffff9d8188741880 x1862023066544256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767493.813183:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066544640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767493.813207:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767493.813209:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767493.813211:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767493.813214:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4325 00000100:00080000:3.0:1775767493.813216:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767493.813219:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066544256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767493.813221:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767493.813227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767493.813239:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066544768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767493.813240:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767493.813252:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767493.813256:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066544256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767493.813276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767493.813295:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767493.813303:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767493/real 1775767493] req@ffff9d8188743800 x1862023066544384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767493.813312:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066544384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767493.813317:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767493.813320:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767493.813322:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767493.813325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767493.813335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767493.813341:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767493/real 1775767493] req@ffff9d8188741f80 x1862023066544512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767493.813352:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066544512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767493.813357:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767493.813359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767493.813361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767493.813364:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767493.813374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767493.813378:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767493/real 1775767493] req@ffff9d8188740000 x1862023066544640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767493.813385:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066544640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767493.813394:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767493.813398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767493.813401:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767493.813407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767493.813425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767493.813436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767493/real 1775767493] req@ffff9d8188740700 x1862023066544768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767493.813449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066544768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767493.813461:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767493.813464:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767493.813466:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066544768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767494.453135:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767494.836994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767495.477040:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767495.860999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767496.501113:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767496.884962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767497.525096:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767497.908937:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767498.549064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767498.932963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767498.932964:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767498.932974:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767498.932978:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767498.932986:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4330 00000100:00080000:3.0:1775767498.932995:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767498.933000:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767498.933056:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066544896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767498.933079:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767498.933082:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767498.933083:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767498.933086:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4330 00000100:00080000:3.0:1775767498.933089:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767498.933092:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767498.933100:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767498.933110:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066545024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767498.933118:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767498.933119:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066544896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767498.933121:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767498.933124:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767498.933131:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4330 00000100:00080000:3.0:1775767498.933133:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767498.933135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767498.933138:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767498.933148:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066545152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767498.933155:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767498.933157:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767498.933158:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767498.933160:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4330 00000100:00080000:3.0:1775767498.933174:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767498.933176:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767498.933180:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767498.933184:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767498.933198:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066545280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767498.933205:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767498.933207:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767498.933208:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767498.933210:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4330 00000100:00080000:3.0:1775767498.933213:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767498.933216:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767498.933219:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767498.933227:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767498/real 1775767498] req@ffff9d8188742a00 x1862023066544896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767498.933230:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066545408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767498.933264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066544896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767498.933272:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767498.933276:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767498.933279:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066544896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767498.933296:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767498.933312:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767498.933320:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767498/real 1775767498] req@ffff9d8188741c00 x1862023066545024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767498.933328:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066545024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767498.933334:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767498.933336:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767498.933338:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767498.933342:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767498.933352:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767498.933357:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767498/real 1775767498] req@ffff9d8188740380 x1862023066545152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767498.933365:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066545152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767498.933370:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767498.933372:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767498.933373:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767498.933377:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767498.933387:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767498.933392:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767498/real 1775767498] req@ffff9d8188740700 x1862023066545280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767498.933400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066545280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767498.933410:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767498.933412:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767498.933414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767498.933418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767498.933427:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767498.933432:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767498/real 1775767498] req@ffff9d8188740000 x1862023066545408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767498.933440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066545408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767498.933449:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767498.933453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767498.933455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767499.573068:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767499.956964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767500.597119:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767500.981015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767501.621073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767502.004970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767502.645086:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767503.029026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767503.669106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767504.052997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767504.052997:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767504.053007:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767504.053012:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767504.053025:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4335 00000100:00080000:3.0:1775767504.053039:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767504.053048:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767504.053101:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066545536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767504.053129:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767504.053132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767504.053134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767504.053137:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4335 00000100:00080000:3.0:1775767504.053141:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767504.053145:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767504.053156:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767504.053163:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066545536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767504.053169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066545664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767504.053177:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767504.053180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767504.053181:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767504.053184:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4335 00000100:00080000:3.0:1775767504.053186:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767504.053190:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767504.053194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767504.053204:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066545792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767504.053210:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767504.053213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767504.053214:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767504.053217:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4335 00000100:00080000:3.0:1775767504.053229:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767504.053231:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767504.053234:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767504.053241:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767504.053246:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066545920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767504.053253:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767504.053256:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767504.053257:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767504.053260:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4335 00000100:00080000:3.0:1775767504.053265:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767504.053267:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767504.053270:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767504.053277:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066546048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767504.053299:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767504/real 1775767504] req@ffff9d8188741f80 x1862023066545536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767504.053315:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066545536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767504.053324:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767504.053327:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767504.053330:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066545536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767504.053351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767504.053366:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767504.053375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767504/real 1775767504] req@ffff9d8188743800 x1862023066545664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767504.053383:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066545664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767504.053389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767504.053391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767504.053393:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767504.053397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767504.053407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767504.053412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767504/real 1775767504] req@ffff9d8188741880 x1862023066545792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767504.053419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066545792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767504.053429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767504.053431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767504.053434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767504.053437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767504.053447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767504.053452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767504/real 1775767504] req@ffff9d8188742680 x1862023066545920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767504.053460:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066545920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767504.053466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767504.053467:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767504.053469:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066545920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767504.053472:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767504.053482:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767504.053488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767504/real 1775767504] req@ffff9d8188742300 x1862023066546048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767504.053495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066546048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767504.053500:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767504.053502:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767504.053504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767504.693096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767505.077053:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767505.717125:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767506.101029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767506.741040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767507.125072:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767507.765091:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767508.149065:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767508.789111:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767509.173000:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767509.173009:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767509.173013:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767509.173021:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4340 00000100:00080000:0.0:1775767509.173031:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767509.173039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767509.173043:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767509.173080:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066546176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767509.173103:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767509.173106:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767509.173107:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767509.173110:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4340 00000100:00080000:0.0:1775767509.173112:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767509.173114:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767509.173132:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767509.173140:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066546304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767509.173152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767509.173156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1775767509.173156:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066546176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767509.173158:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767509.173165:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4340 00000100:00080000:0.0:1775767509.173168:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767509.173170:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767509.173173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767509.173181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066546432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767509.173187:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767509.173189:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767509.173190:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767509.173193:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4340 00000100:00080000:0.0:1775767509.173195:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767509.173197:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767509.173201:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767509.173207:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066546560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767509.173213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767509.173215:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767509.173216:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767509.173218:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4340 00000100:00080000:0.0:1775767509.173221:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767509.173224:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767509.173227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775767509.173230:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767509.173237:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066546688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767509.173286:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767509/real 1775767509] req@ffff9d81901afb80 x1862023066546176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767509.173305:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066546176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767509.173315:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767509.173320:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767509.173324:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066546176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767509.173358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767509.173382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767509.173395:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767509/real 1775767509] req@ffff9d81901aed80 x1862023066546304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767509.173408:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066546304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767509.173417:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767509.173420:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767509.173423:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767509.173430:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767509.173449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767509.173459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767509/real 1775767509] req@ffff9d81901af100 x1862023066546432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767509.173472:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066546432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767509.173481:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767509.173484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767509.173487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767509.173493:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767509.173513:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767509.173523:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767509/real 1775767509] req@ffff9d81901ae680 x1862023066546560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767509.173535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066546560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767509.173545:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767509.173549:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767509.173552:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767509.173558:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767509.173578:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767509.173588:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767509/real 1775767509] req@ffff9d81901ad180 x1862023066546688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767509.173599:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066546688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767509.173609:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767509.173612:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767509.173615:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767509.813028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767510.197042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767510.837087:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767511.221030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767511.860998:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767512.245047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767512.885048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767513.268983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767513.909106:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767514.293002:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767514.293012:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767514.293016:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767514.293022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767514.293026:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4345 00000100:00080000:0.0:1775767514.293038:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767514.293046:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767514.293102:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066546816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767514.293133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767514.293137:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767514.293140:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767514.293145:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4345 00000100:00080000:0.0:1775767514.293151:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767514.293157:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767514.293168:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066546816:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775767514.293169:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767514.293188:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066546944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767514.293199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767514.293202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767514.293204:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767514.293208:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4345 00000100:00080000:0.0:1775767514.293220:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767514.293223:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767514.293227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767514.293231:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767514.293241:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066547072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767514.293249:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767514.293252:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767514.293253:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767514.293256:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4345 00000100:00080000:0.0:1775767514.293259:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767514.293261:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767514.293266:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767514.293277:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066547200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767514.293285:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767514.293289:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767514.293291:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767514.293291:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767514/real 1775767514] req@ffff9d81901ad180 x1862023066546816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767514.293300:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4345 00000100:00080000:0.0:1775767514.293306:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767514.293312:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767514.293318:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767514.293333:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066547328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767514.293337:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066546816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767514.293367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767514.293373:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767514.293379:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066546816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767514.293406:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767514.293443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767514.293458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767514/real 1775767514] req@ffff9d81901ae680 x1862023066546944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767514.293470:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066546944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767514.293482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767514.293492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767514.293495:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066546944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767514.293500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767514.293514:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767514.293522:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767514/real 1775767514] req@ffff9d81901af100 x1862023066547072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767514.293531:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066547072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767514.293537:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767514.293539:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767514.293542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767514.293546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767514.293558:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767514.293564:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767514/real 1775767514] req@ffff9d81901aed80 x1862023066547200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767514.293572:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066547200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767514.293580:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767514.293582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767514.293585:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767514.293589:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767514.293600:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767514.293606:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767514/real 1775767514] req@ffff9d81901afb80 x1862023066547328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767514.293614:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066547328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767514.293620:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767514.293622:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767514.293624:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767514.933076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767515.316990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767515.957056:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767516.341087:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767516.981112:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767517.364993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767518.005076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767518.389009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767519.029063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775767519.412962:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767519.412969:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767519.412973:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767519.412973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767519.412982:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4350 00000100:00080000:0.0:1775767519.412991:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767519.412997:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767519.413032:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066547456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767519.413053:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767519.413056:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767519.413057:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767519.413060:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4350 00000100:00080000:0.0:1775767519.413064:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767519.413079:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767519.413088:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767519.413090:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066547456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767519.413100:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066547584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767519.413109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767519.413111:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767519.413113:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767519.413116:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4350 00000100:00080000:0.0:1775767519.413119:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767519.413121:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767519.413124:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767519.413133:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066547712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767519.413139:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767519.413142:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767519.413143:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767519.413146:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4350 00000800:00000100:3.0:1775767519.413150:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775767519.413151:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767519.413155:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767519.413160:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767519.413168:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066547840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767519.413181:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767519.413183:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767519.413185:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767519.413189:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4350 00000100:00080000:0.0:1775767519.413193:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767519.413193:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767519/real 1775767519] req@ffff9d81901afb80 x1862023066547456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767519.413200:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767519.413208:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767519.413220:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066547968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767519.413222:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066547456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767519.413273:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767519.413278:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767519.413282:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066547456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767519.413308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767519.413330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767519.413339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767519/real 1775767519] req@ffff9d81901aed80 x1862023066547584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767519.413350:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066547584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767519.413356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767519.413359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767519.413361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767519.413366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767519.413378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767519.413384:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767519/real 1775767519] req@ffff9d81901af100 x1862023066547712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767519.413393:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066547712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767519.413399:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767519.413401:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767519.413403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767519.413407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767519.413419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767519.413425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767519/real 1775767519] req@ffff9d81901ae680 x1862023066547840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767519.413434:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066547840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767519.413441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767519.413443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767519.413445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767519.413449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767519.413460:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767519.413466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767519/real 1775767519] req@ffff9d81901ad180 x1862023066547968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767519.413474:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066547968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767519.413480:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767519.413483:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767519.413485:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066547968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767520.053089:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767520.437069:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767521.077255:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767521.461056:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767522.101036:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767522.484973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767523.125048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767523.508962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767524.149122:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767524.533028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767524.533028:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767524.533044:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767524.533048:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767524.533058:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4356 00000100:00080000:2.0:1775767524.533074:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767524.533080:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767524.533118:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066548096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767524.533143:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767524.533147:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767524.533148:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767524.533150:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4356 00000100:00080000:2.0:1775767524.533153:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767524.533156:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767524.533163:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767524.533173:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066548224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767524.533180:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767524.533183:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767524.533184:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767524.533188:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4356 00000100:00080000:2.0:1775767524.533190:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767524.533192:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767524.533196:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767524.533204:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066548096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767524.533210:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066548352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767524.533226:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767524.533228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767524.533229:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767524.533231:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4356 00000100:00080000:2.0:1775767524.533233:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767524.533236:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767524.533240:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767524.533264:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066548480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767524.533282:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767524.533285:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767524.533287:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767524.533291:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4356 00000100:00080000:2.0:1775767524.533296:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767524.533299:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767524.533304:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767524.533314:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066548608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767524.533357:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767524.533394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767524.533411:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767524.533426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767525.173026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767525.557034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767526.197049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767526.581048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767527.221124:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767527.605140:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775767527.605214:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775767527.605263:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775767527.605280:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767527.605320:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767527.605330:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767527.605341:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767527.605354:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775767527.605367:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767524/real 1775767527] req@ffff9d81984a3b80 x1862023066548352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767527.605408:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066548352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767527.605421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767527.605427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767527.605432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767527.605441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767524/real 1775767527] req@ffff9d81984a0000 x1862023066548480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767527.605458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066548480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767527.605468:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767527.605471:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767527.605474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767527.605481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767524/real 1775767527] req@ffff9d81984a2680 x1862023066548608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767527.605493:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066548608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767527.605502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767527.605506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767527.605509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548608:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767527.605515:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767524/real 1775767527] req@ffff9d81984a0e00 x1862023066548096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767527.605529:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066548096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767527.605537:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767527.605540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767527.605543:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066548096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767527.605550:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767524/real 1775767527] req@ffff9d81984a1f80 x1862023066548224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767527.605563:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066548224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767527.605571:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767527.605575:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767527.605579:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548224:192.168.202.105@tcp:38:kworker.0 00010000:00010000:0.0:1775767528.245009:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767528.629064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767529.268999:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767529.652984:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767529.652992:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767529.652996:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767529.653005:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4361 00000100:00080000:3.0:1775767529.653013:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767529.653019:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767529.653028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775767529.653052:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066548736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767529.653075:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767529.653077:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767529.653079:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767529.653082:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4361 00000100:00080000:3.0:1775767529.653085:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767529.653087:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767529.653096:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767529.653098:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066548736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767529.653110:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066548864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767529.653119:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767529.653121:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767529.653123:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767529.653126:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4361 00000100:00080000:3.0:1775767529.653130:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767529.653132:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767529.653135:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767529.653146:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066548992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767529.653159:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767529.653170:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767529.653174:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767529.653177:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767529.653185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4361 00000100:00080000:3.0:1775767529.653190:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767529.653197:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767529.653203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767529.653218:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767529/real 1775767529] req@ffff9d8188740e00 x1862023066548736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767529.653218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066549120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767529.653254:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767529.653257:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767529.653259:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767529.653263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066548736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767529.653267:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4361 00000100:00080000:3.0:1775767529.653274:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767529.653283:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775767529.653284:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775767529.653290:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767529.653293:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767529.653300:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066548736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767529.653308:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066549248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767529.653327:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767529.653353:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767529.653367:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767529/real 1775767529] req@ffff9d8188741180 x1862023066548864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767529.653381:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066548864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767529.653390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767529.653393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767529.653396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767529.653402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767529.653422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767529.653433:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767529/real 1775767529] req@ffff9d8188743100 x1862023066548992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767529.653449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066548992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767529.653457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767529.653460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767529.653463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066548992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767529.653469:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767529.653489:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767529.653500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767529/real 1775767529] req@ffff9d8188743480 x1862023066549120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767529.653512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066549120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767529.653522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767529.653525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767529.653528:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767529.653533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767529.653551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767529.653560:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767529/real 1775767529] req@ffff9d8188743b80 x1862023066549248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767529.653571:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066549248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767529.653583:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767529.653587:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767529.653589:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767530.293136:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767530.677101:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767531.317015:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767531.700974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767532.341122:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767532.724975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767533.365070:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767533.748976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767534.389067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767534.772954:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767534.772956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767534.772964:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767534.772968:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767534.772976:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4366 00000100:00080000:3.0:1775767534.772985:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767534.772991:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767534.773040:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066549376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767534.773065:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767534.773068:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767534.773070:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767534.773073:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4366 00000100:00080000:3.0:1775767534.773076:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767534.773079:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767534.773088:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767534.773099:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066549376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767534.773101:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066549504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767534.773112:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767534.773115:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767534.773116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767534.773120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4366 00000100:00080000:3.0:1775767534.773122:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767534.773125:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767534.773128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767534.773137:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066549632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767534.773143:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767534.773147:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767534.773148:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767534.773151:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4366 00000100:00080000:3.0:1775767534.773154:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767534.773156:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767534.773160:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767534.773161:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767534.773173:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066549760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767534.773180:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767534.773183:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767534.773185:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767534.773188:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4366 00000100:00080000:3.0:1775767534.773191:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767534.773194:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767534.773198:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767534.773206:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066549888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767534.773245:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767534/real 1775767534] req@ffff9d8188741500 x1862023066549376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767534.773267:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066549376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767534.773279:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767534.773283:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767534.773286:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066549376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767534.773305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767534.773323:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767534.773332:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767534/real 1775767534] req@ffff9d8188740a80 x1862023066549504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767534.773341:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066549504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767534.773346:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767534.773348:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767534.773351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767534.773354:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767534.773364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767534.773370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767534/real 1775767534] req@ffff9d8188742d80 x1862023066549632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767534.773377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066549632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767534.773382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767534.773384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767534.773385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767534.773389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767534.773400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767534.773405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767534/real 1775767534] req@ffff9d8188743b80 x1862023066549760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767534.773412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066549760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767534.773419:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767534.773421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767534.773422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767534.773426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767534.773435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767534.773440:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767534/real 1775767534] req@ffff9d8188743480 x1862023066549888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767534.773446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066549888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767534.773452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767534.773453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767534.773455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066549888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767535.413030:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767535.796995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767536.437062:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767536.820999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767537.461044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767537.844993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767538.485014:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767538.868943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767539.509032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767539.892957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767539.892996:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767539.893005:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767539.893009:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767539.893017:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4371 00000100:00080000:1.0:1775767539.893026:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767539.893031:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767539.893065:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066550016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767539.893091:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767539.893094:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767539.893095:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767539.893098:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4371 00000100:00080000:1.0:1775767539.893101:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767539.893104:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767539.893112:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767539.893122:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066550144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767539.893132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066550016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767539.893135:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767539.893146:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767539.893147:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767539.893151:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4371 00000100:00080000:1.0:1775767539.893153:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767539.893155:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767539.893159:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767539.893173:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066550272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767539.893179:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767539.893181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767539.893183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767539.893185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4371 00000100:00080000:1.0:1775767539.893188:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767539.893196:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767539.893199:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767539.893205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066550400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767539.893216:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767539.893219:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767539.893220:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767539.893229:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767539.893233:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4371 00000100:00080000:1.0:1775767539.893237:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767539.893240:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767539.893244:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767539.893258:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066550528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767539.893292:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767539/real 1775767539] req@ffff9d8190550e00 x1862023066550016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767539.893311:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066550016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767539.893321:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767539.893325:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767539.893328:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066550016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767539.893351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767539.893369:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767539.893377:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767539/real 1775767539] req@ffff9d8190551880 x1862023066550144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767539.893386:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066550144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767539.893392:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767539.893394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767539.893396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767539.893401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767539.893411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767539.893417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767539/real 1775767539] req@ffff9d8190551500 x1862023066550272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767539.893424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066550272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767539.893434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767539.893437:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767539.893440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767539.893443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767539.893455:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767539.893460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767539/real 1775767539] req@ffff9d8190553100 x1862023066550400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767539.893467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066550400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767539.893474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767539.893476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767539.893477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767539.893481:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767539.893499:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767539.893506:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767539/real 1775767539] req@ffff9d8190553480 x1862023066550528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767539.893512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066550528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767539.893519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767539.893521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767539.893524:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767540.533059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767540.916957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767541.557065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767541.940982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767542.581036:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767542.964955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767543.605076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767543.988988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767544.629040:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767545.012985:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767545.012993:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767545.012997:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767545.013007:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4376 00000100:00100000:0.0:1775767545.013012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767545.013017:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767545.013030:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767545.013076:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066550656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767545.013100:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767545.013105:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767545.013107:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767545.013110:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4376 00000100:00080000:1.0:1775767545.013115:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767545.013120:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767545.013127:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066550656:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775767545.013142:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767545.013160:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066550784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767545.013170:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767545.013172:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767545.013174:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767545.013177:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4376 00000100:00080000:1.0:1775767545.013195:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767545.013197:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767545.013201:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767545.013207:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767545.013211:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066550912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767545.013219:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767545.013222:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767545.013224:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767545.013227:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4376 00000100:00080000:1.0:1775767545.013230:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767545.013233:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767545.013238:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767545.013252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066551040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767545.013261:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767545.013264:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767545.013265:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767545.013272:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4376 00000100:00080000:1.0:1775767545.013277:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767545.013281:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767545/real 1775767545] req@ffff9d8190553480 x1862023066550656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767545.013285:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767545.013298:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767545.013320:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066551168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767545.013327:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066550656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767545.013354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767545.013363:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767545.013368:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066550656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767545.013399:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767545.013425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767545.013438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767545/real 1775767545] req@ffff9d8190553100 x1862023066550784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767545.013452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066550784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767545.013471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767545.013475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767545.013479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767545.013485:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767545.013509:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767545.013517:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767545/real 1775767545] req@ffff9d8190551500 x1862023066550912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767545.013525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066550912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767545.013531:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767545.013534:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767545.013536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066550912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767545.013541:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767545.013553:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767545.013559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767545/real 1775767545] req@ffff9d8190551880 x1862023066551040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767545.013567:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066551040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767545.013596:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767545.013599:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767545.013603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767545.013608:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767545.013623:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767545.013631:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767545/real 1775767545] req@ffff9d8190550e00 x1862023066551168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767545.013639:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066551168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767545.013645:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767545.013647:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767545.013650:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767545.653077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767546.037025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767546.677074:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767547.061021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767547.701070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767548.085051:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767548.725049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767549.108977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767549.749061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767550.133009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767550.133045:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767550.133053:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767550.133056:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767550.133066:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4381 00000100:00080000:1.0:1775767550.133077:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767550.133084:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767550.133126:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066551296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767550.133151:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767550.133154:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767550.133156:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767550.133160:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4381 00000100:00080000:1.0:1775767550.133163:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767550.133168:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767550.133178:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767550.133194:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066551424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767550.133196:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066551296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767550.133213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767550.133216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767550.133219:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767550.133223:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4381 00000100:00080000:1.0:1775767550.133227:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767550.133231:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767550.133237:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767550.133249:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066551552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767550.133257:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767550.133258:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767550.133262:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767550.133264:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767550.133268:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4381 00000100:00080000:1.0:1775767550.133272:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767550.133276:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767550.133280:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767550.133289:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066551680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767550.133296:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767550.133298:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767550.133300:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767550.133302:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4381 00000100:00100000:0.0:1775767550.133306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767550/real 1775767550] req@ffff9d8190550e00 x1862023066551296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767550.133307:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767550.133314:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767550.133320:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767550.133329:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066551808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767550.133333:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066551296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767550.133355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767550.133359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767550.133362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066551296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767550.133379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767550.133393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767550.133401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767550/real 1775767550] req@ffff9d8190551880 x1862023066551424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767550.133410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066551424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767550.133418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767550.133422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767550.133425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767550.133431:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767550.133451:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767550.133462:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767550/real 1775767550] req@ffff9d8190551500 x1862023066551552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767550.133476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066551552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767550.133494:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767550.133496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767550.133499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767550.133504:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767550.133520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767550.133563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767550/real 1775767550] req@ffff9d8190553100 x1862023066551680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767550.133576:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066551680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767550.133586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767550.133590:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767550.133593:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767550.133599:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767550.133617:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767550.133624:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767550/real 1775767550] req@ffff9d8190553480 x1862023066551808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767550.133632:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066551808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767550.133638:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767550.133640:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767550.133642:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066551808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767550.773054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767551.156948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767551.797072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767552.180962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767552.821059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767553.204954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767553.845078:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767554.229005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767554.869011:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767555.252976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767555.253019:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767555.253029:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767555.253034:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767555.253044:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4386 00000100:00080000:2.0:1775767555.253056:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767555.253063:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767555.253101:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066551936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767555.253139:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767555.253143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767555.253145:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767555.253149:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4386 00000100:00080000:2.0:1775767555.253153:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767555.253157:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767555.253166:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767555.253179:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066552064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767555.253191:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066551936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767555.253193:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767555.253200:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767555.253202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767555.253208:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4386 00000100:00080000:2.0:1775767555.253212:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767555.253216:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767555.253221:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767555.253234:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066552192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767555.253244:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767555.253248:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767555.253249:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767555.253254:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775767555.253258:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4386 00000100:00080000:2.0:1775767555.253264:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767555.253269:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767555.253274:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767555.253287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066552320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767555.253298:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767555.253302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767555/real 1775767555] req@ffff9d81984a1180 x1862023066551936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767555.253303:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767555.253307:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767555.253317:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4386 00000100:00080000:2.0:1775767555.253322:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767555.253328:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066551936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767555.253329:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767555.253337:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767555.253347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767555.253355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:2.0:1775767555.253355:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066552448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767555.253363:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066551936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767555.253396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767555.253420:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767555.253430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767555/real 1775767555] req@ffff9d81984a1880 x1862023066552064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767555.253439:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066552064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767555.253444:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767555.253447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767555.253449:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767555.253453:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767555.253465:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767555.253470:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767555/real 1775767555] req@ffff9d81984a3800 x1862023066552192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767555.253500:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066552192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767555.253506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767555.253508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767555.253510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767555.253514:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767555.253528:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767555.253535:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767555/real 1775767555] req@ffff9d81984a3100 x1862023066552320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767555.253543:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066552320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767555.253549:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767555.253551:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767555.253553:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767555.253556:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767555.253566:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767555.253571:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767555/real 1775767555] req@ffff9d81984a1500 x1862023066552448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767555.253578:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066552448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767555.253588:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767555.253590:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767555.253592:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767555.893033:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767556.277013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767556.917061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767557.301019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767557.941110:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767558.324972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767558.965030:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767559.348986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767559.989052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767560.372950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767560.372980:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767560.372985:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767560.372988:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767560.372996:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4391 00000100:00080000:2.0:1775767560.373005:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767560.373011:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767560.373045:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066552576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767560.373072:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767560.373076:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767560.373082:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767560.373084:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4391 00000100:00080000:2.0:1775767560.373087:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767560.373089:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767560.373097:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767560.373108:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066552704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767560.373116:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767560.373119:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767560.373122:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066552576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767560.373122:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767560.373127:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4391 00000100:00080000:2.0:1775767560.373131:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767560.373133:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767560.373136:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767560.373145:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066552832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767560.373151:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767560.373154:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767560.373156:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767560.373159:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4391 00000100:00080000:2.0:1775767560.373170:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767560.373172:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767560.373177:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767560.373180:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767560.373189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066552960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767560.373197:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767560.373200:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767560.373201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767560.373203:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4391 00000100:00080000:2.0:1775767560.373205:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767560.373207:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767560.373211:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767560.373218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066553088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767560.373230:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767560/real 1775767560] req@ffff9d81984a0a80 x1862023066552576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767560.373252:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066552576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767560.373263:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767560.373267:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767560.373271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066552576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767560.373290:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767560.373308:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767560.373320:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767560/real 1775767560] req@ffff9d81984a1500 x1862023066552704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767560.373333:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066552704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767560.373348:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767560.373352:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767560.373356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767560.373362:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767560.373382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767560.373390:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767560/real 1775767560] req@ffff9d81984a3100 x1862023066552832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767560.373398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066552832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767560.373403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767560.373414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767560.373417:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767560.373421:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767560.373449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767560.373456:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767560/real 1775767560] req@ffff9d81984a3800 x1862023066552960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767560.373464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066552960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767560.373472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767560.373474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767560.373476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066552960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767560.373480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767560.373491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767560.373496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767560/real 1775767560] req@ffff9d81984a1880 x1862023066553088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767560.373504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066553088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767560.373509:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767560.373510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767560.373512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767561.013065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767561.396978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767562.037066:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767562.420972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767563.061065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767563.444952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767564.085060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767564.468935:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767565.109015:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767565.492975:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767565.492985:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1775767565.492985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767565.492989:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767565.493000:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4397 00000100:00080000:1.0:1775767565.493011:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767565.493018:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767565.493058:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066553216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767565.493087:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767565.493092:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767565.493094:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767565.493098:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4397 00000100:00080000:1.0:1775767565.493102:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767565.493106:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767565.493116:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767565.493131:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066553344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767565.493145:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066553216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767565.493145:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767565.493156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767565.493158:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767565.493163:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4397 00000100:00080000:1.0:1775767565.493168:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767565.493172:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767565.493178:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767565.493192:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066553472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767565.493201:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767565.493205:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767565.493206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767565.493213:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767565.493216:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4397 00000100:00080000:1.0:1775767565.493221:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767565.493226:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767565.493232:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767565.493245:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066553600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767565.493256:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767565.493260:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767565.493263:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767565.493264:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767565/real 1775767565] req@ffff9d8190553480 x1862023066553216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767565.493271:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4397 00000100:00080000:1.0:1775767565.493279:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767565.493285:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767565.493290:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767565.493295:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066553216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767565.493307:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066553728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767565.493309:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767565.493319:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767565.493326:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066553216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767565.493351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767565.493373:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767565.493401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767565/real 1775767565] req@ffff9d8190553100 x1862023066553344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767565.493413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066553344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767565.493421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767565.493424:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767565.493428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767565.493434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767565.493453:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767565.493464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767565/real 1775767565] req@ffff9d8190551500 x1862023066553472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767565.493475:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066553472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767565.493483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767565.493487:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767565.493491:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767565.493496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767565.493514:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767565.493524:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767565/real 1775767565] req@ffff9d8190551880 x1862023066553600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767565.493536:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066553600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767565.493546:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767565.493549:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767565.493553:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767565.493559:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767565.493578:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767565.493589:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767565/real 1775767565] req@ffff9d8190550e00 x1862023066553728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767565.493601:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066553728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767565.493610:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767565.493614:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767565.493617:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767566.133014:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767566.516941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767567.157059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767567.540963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767568.181039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767568.564938:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767569.205049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767569.588973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767570.229019:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767570.612960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767570.613020:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767570.613026:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767570.613030:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767570.613037:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4402 00000100:00080000:2.0:1775767570.613046:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767570.613051:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767570.613085:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066553856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767570.613111:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767570.613114:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767570.613115:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767570.613122:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4402 00000100:00080000:2.0:1775767570.613125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767570.613128:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767570.613154:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767570.613172:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066553984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767570.613187:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767570.613192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767570.613195:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767570.613200:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066553856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767570.613205:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4402 00000100:00080000:2.0:1775767570.613231:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767570.613236:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767570.613241:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767570.613254:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066554112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767570.613264:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767570.613267:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767570.613269:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767570.613273:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4402 00000100:00080000:2.0:1775767570.613277:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767570.613282:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775767570.613286:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767570.613293:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767570.613307:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066554240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767570.613323:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767570/real 1775767570] req@ffff9d81984a1180 x1862023066553856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.613324:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767570.613344:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767570.613353:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767570.613362:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4402 00000100:00080000:2.0:1775767570.613366:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767570.613372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066553856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767570.613372:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767570.613380:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767570.613387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767570.613393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:2.0:1775767570.613396:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066554368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767570.613400:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066553856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767570.613427:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767570.613445:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767570.613453:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767570/real 1775767570] req@ffff9d81984a1f80 x1862023066553984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767570.613461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066553984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767570.613467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767570.613469:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767570.613471:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066553984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767570.613474:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767570.613484:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767570.613489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767570/real 1775767570] req@ffff9d81984a0e00 x1862023066554112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767570.613497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066554112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767570.613506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767570.613508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767570.613509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767570.613513:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767570.613522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767570.613527:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767570/real 1775767570] req@ffff9d81984a2680 x1862023066554240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767570.613534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066554240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767570.613541:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767570.613543:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767570.613545:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767570.613548:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767570.613557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767570.613561:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767570/real 1775767570] req@ffff9d81984a0000 x1862023066554368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767570.613568:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066554368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767570.613573:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767570.613575:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767570.613577:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767571.253046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767571.637024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767572.277016:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767572.660974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767573.301056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767573.684982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767574.325114:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767574.708992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767575.349046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767575.732971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767575.733055:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767575.733061:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767575.733064:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767575.733072:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4407 00000100:00080000:2.0:1775767575.733091:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767575.733098:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767575.733133:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066554496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767575.733157:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767575.733160:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767575.733161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767575.733164:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4407 00000100:00080000:2.0:1775767575.733166:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767575.733169:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767575.733177:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767575.733185:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066554624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767575.733193:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767575.733195:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767575.733197:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767575.733200:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4407 00000100:00080000:2.0:1775767575.733202:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767575.733204:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767575.733207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767575.733211:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066554496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767575.733217:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066554752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767575.733224:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767575.733226:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767575.733228:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767575.733230:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4407 00000100:00080000:2.0:1775767575.733232:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767575.733234:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767575.733237:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767575.733246:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066554880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767575.733252:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767575.733254:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767575.733256:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767575.733259:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4407 00000100:00080000:2.0:1775767575.733262:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767575.733264:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767575.733267:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767575.733269:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767575.733291:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066555008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767575.733325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767575/real 1775767575] req@ffff9d81984a3b80 x1862023066554496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767575.733340:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066554496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767575.733347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767575.733350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767575.733353:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066554496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767575.733359:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767575.733374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767575.733382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767575/real 1775767575] req@ffff9d81984a3480 x1862023066554624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767575.733390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066554624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767575.733396:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767575.733400:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767575.733403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767575.733426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767575.733448:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767575.733460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767575/real 1775767575] req@ffff9d81984a2d80 x1862023066554752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767575.733471:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066554752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767575.733476:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767575.733479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767575.733481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767575.733485:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767575.733499:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767575.733505:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767575/real 1775767575] req@ffff9d81984a0700 x1862023066554880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767575.733512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066554880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767575.733532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767575.733534:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767575.733536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066554880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767575.733540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767575.733551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767575.733557:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767575/real 1775767575] req@ffff9d81984a2a00 x1862023066555008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767575.733565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066555008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767575.733571:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767575.733573:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767575.733575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767576.373027:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767576.756993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767577.397060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767577.780955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767578.421791:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767578.804993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767579.445112:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767579.828935:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767580.469003:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767580.852997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767580.853027:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767580.853034:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767580.853037:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767580.853045:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4412 00000100:00080000:0.0:1775767580.853053:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767580.853058:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767580.853093:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066555136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767580.853120:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767580.853122:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767580.853124:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767580.853126:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4412 00000100:00080000:0.0:1775767580.853128:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767580.853131:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767580.853138:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767580.853146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066555264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767580.853153:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767580.853156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767580.853157:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767580.853160:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4412 00000100:00080000:0.0:1775767580.853162:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767580.853164:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767580.853166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767580.853172:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066555392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767580.853180:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767580.853182:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767580.853184:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767580.853188:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4412 00000100:00080000:0.0:1775767580.853190:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767580.853191:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066555136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767580.853193:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767580.853202:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767580.853210:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066555520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767580.853216:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767580.853219:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767580.853220:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767580.853223:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4412 00000100:00080000:0.0:1775767580.853241:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767580.853246:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767580.853252:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767580.853262:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066555648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767580.853279:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767580.853317:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767580/real 1775767580] req@ffff9d81901ad180 x1862023066555136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767580.853331:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066555136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767580.853339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767580.853342:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767580.853345:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066555136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767580.853350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767580.853365:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767580.853373:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767580/real 1775767580] req@ffff9d81901ae680 x1862023066555264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767580.853385:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066555264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767580.853390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767580.853392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767580.853394:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767580.853415:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767580.853434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767580.853446:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767580/real 1775767580] req@ffff9d81901af100 x1862023066555392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767580.853457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066555392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767580.853465:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767580.853468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767580.853471:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767580.853476:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767580.853494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767580.853504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767580/real 1775767580] req@ffff9d81901aed80 x1862023066555520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767580.853515:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066555520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767580.853529:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767580.853533:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767580.853535:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767580.853540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767580.853557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767580.853566:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767580/real 1775767580] req@ffff9d81901afb80 x1862023066555648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767580.853577:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066555648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767580.853585:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767580.853588:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767580.853591:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767581.493020:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767581.876967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767582.517012:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767582.900936:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767583.541058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767583.924982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767584.565013:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767584.948957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767585.589067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767585.972949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767585.972950:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767585.972958:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767585.972962:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767585.972970:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4417 00000100:00080000:0.0:1775767585.972980:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767585.972986:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767585.973020:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066555776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767585.973044:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767585.973047:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767585.973049:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767585.973053:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4417 00000100:00080000:0.0:1775767585.973057:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767585.973061:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767585.973070:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767585.973086:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066555904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767585.973091:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066555776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767585.973108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767585.973110:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767585.973112:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767585.973116:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4417 00000100:00080000:0.0:1775767585.973119:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767585.973122:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767585.973127:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767585.973139:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066556032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767585.973148:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767585.973151:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775767585.973156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767585.973159:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767585.973166:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4417 00000100:00080000:0.0:1775767585.973171:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767585.973176:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767585.973195:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767585.973203:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767585/real 1775767585] req@ffff9d81901afb80 x1862023066555776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767585.973218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066556160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767585.973221:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066555776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767585.973244:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767585.973252:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767585.973255:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767585.973259:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767585.973261:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4417 00000100:00080000:3.0:1775767585.973265:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:0.0:1775767585.973266:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767585.973271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066555776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767585.973273:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767585.973280:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767585.973289:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066556288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767585.973297:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767585.973316:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767585.973326:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767585/real 1775767585] req@ffff9d81901aed80 x1862023066555904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767585.973334:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066555904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767585.973339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767585.973341:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767585.973343:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066555904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767585.973347:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767585.973363:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767585.973369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767585/real 1775767585] req@ffff9d81901af100 x1862023066556032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767585.973376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066556032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767585.973381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767585.973383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767585.973386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767585.973390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767585.973400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767585.973405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767585/real 1775767585] req@ffff9d81901ae680 x1862023066556160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767585.973412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066556160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767585.973418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767585.973420:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767585.973422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767585.973431:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767585.973440:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767585.973445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767585/real 1775767585] req@ffff9d81901ad180 x1862023066556288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767585.973452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066556288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767585.973457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767585.973459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767585.973460:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767586.613014:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767586.997009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767587.637085:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767588.021049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767588.661071:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767589.044983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767589.685044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767590.068951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767590.709088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767591.092997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767591.093030:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767591.093037:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767591.093042:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767591.093050:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4422 00000100:00080000:1.0:1775767591.093061:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767591.093068:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767591.093114:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066556416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767591.093163:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767591.093166:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767591.093168:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767591.093171:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4422 00000100:00080000:1.0:1775767591.093174:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767591.093178:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767591.093189:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767591.093195:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066556416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767591.093201:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066556544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767591.093209:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767591.093212:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767591.093213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767591.093217:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4422 00000100:00080000:1.0:1775767591.093219:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767591.093221:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767591.093226:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767591.093234:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066556672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767591.093241:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767591.093243:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767591.093244:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767591.093247:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4422 00000100:00080000:1.0:1775767591.093251:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767591.093253:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767591.093256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767591.093264:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066556800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767591.093270:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767591.093273:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767591.093274:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767591.093278:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4422 00000100:00080000:1.0:1775767591.093281:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767591.093285:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767591.093291:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767591.093304:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066556928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767591.093323:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767591.093356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767591.093371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767591.093385:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767591.733022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767592.117010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767592.757105:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767593.140991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767593.781010:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767594.164941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767594.165341:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767594.165393:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767594.165413:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767594.165457:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767594.165470:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767594.165480:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767594.165489:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775767594.167251:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767591/real 1775767594] req@ffff9d8190551880 x1862023066556544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767594.167279:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066556544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767594.167291:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767594.167297:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767594.167301:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767594.167310:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767591/real 1775767594] req@ffff9d8190551500 x1862023066556672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767594.167323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066556672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767594.167332:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767594.167336:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767594.167340:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767594.167347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767591/real 1775767594] req@ffff9d8190553100 x1862023066556800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767594.167361:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066556800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767594.167371:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767594.167375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767594.167378:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767594.167386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767591/real 1775767594] req@ffff9d8190553480 x1862023066556928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767594.167397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066556928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767594.167403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767594.167406:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767594.167408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066556928:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767594.167412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767591/real 1775767594] req@ffff9d8190550e00 x1862023066556416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767594.167419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066556416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767594.167425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767594.167427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767594.167429:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066556416:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775767594.805064:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767595.188977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767595.829214:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767596.213143:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775767596.213206:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767596.213215:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775767596.213219:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767596.213233:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4427 00000100:00080000:0.0:1775767596.213243:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767596.213249:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767596.213281:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066557056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767596.213305:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767596.213308:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775767596.213310:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767596.213313:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4427 00000100:00080000:0.0:1775767596.213317:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767596.213331:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767596.213341:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767596.213354:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066557184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767596.213366:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767596.213369:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775767596.213370:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767596.213377:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066557056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775767596.213380:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4427 00000100:00080000:0.0:1775767596.213387:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767596.213392:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775767596.213399:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767596.213412:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066557312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767596.213422:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767596.213426:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775767596.213428:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767596.213432:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4427 00000100:00080000:0.0:1775767596.213445:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767596.213452:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775767596.213452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775767596.213461:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767596.213475:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066557440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767596.213486:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767596.213490:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775767596.213492:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767596.213497:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4427 00000100:00080000:0.0:1775767596.213501:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767596.213505:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767596.213511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767596/real 1775767596] req@ffff9d81901ad180 x1862023066557056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775767596.213512:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767596.213528:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066557568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767596.213538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066557056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767596.213549:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767596.213555:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767596.213560:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066557056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767596.213570:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767596.213595:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767596.213607:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767596/real 1775767596] req@ffff9d81901ae680 x1862023066557184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767596.213620:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066557184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767596.213629:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767596.213633:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767596.213637:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767596.213661:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767596.213683:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767596.213695:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767596/real 1775767596] req@ffff9d81901af100 x1862023066557312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767596.213706:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066557312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767596.213714:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767596.213717:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767596.213720:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767596.213725:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767596.213753:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767596.213763:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767596/real 1775767596] req@ffff9d81901aed80 x1862023066557440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767596.213774:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066557440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767596.213784:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767596.213787:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767596.213789:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767596.213795:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767596.213813:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767596.213825:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767596/real 1775767596] req@ffff9d81901afb80 x1862023066557568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767596.213836:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066557568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767596.213845:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767596.213848:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767596.213996:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767596.853104:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767597.236953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767597.877004:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767598.261093:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767598.901038:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767599.284945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767599.925096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767600.309018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767600.949089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767601.333058:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767601.333059:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767601.333070:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767601.333076:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767601.333088:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4432 00000100:00080000:3.0:1775767601.333102:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767601.333117:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767601.333168:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066557696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767601.333199:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767601.333203:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767601.333205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767601.333210:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4432 00000100:00080000:3.0:1775767601.333214:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767601.333219:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767601.333230:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767601.333243:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066557824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767601.333245:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066557696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767601.333260:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767601.333263:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767601.333264:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767601.333268:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4432 00000100:00080000:3.0:1775767601.333270:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767601.333273:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767601.333277:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767601.333285:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066557952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767601.333292:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767601.333295:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767601.333296:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767601.333299:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4432 00000100:00080000:3.0:1775767601.333301:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767601.333303:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767601.333307:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767601.333313:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066558080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767601.333318:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767601.333320:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767601.333323:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767601.333327:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4432 00000100:00080000:3.0:1775767601.333331:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767601.333335:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767601.333341:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767601.333341:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767601.333356:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066558208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767601.333407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767601/real 1775767601] req@ffff9d8188743100 x1862023066557696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767601.333427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066557696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767601.333439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767601.333443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767601.333448:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066557696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767601.333472:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767601.333493:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767601.333505:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767601/real 1775767601] req@ffff9d8188741180 x1862023066557824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767601.333519:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066557824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767601.333534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767601.333538:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767601.333542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767601.333547:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767601.333566:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767601.333575:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767601/real 1775767601] req@ffff9d8188740e00 x1862023066557952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767601.333588:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066557952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767601.333597:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767601.333601:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767601.333605:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066557952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767601.333613:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767601.333634:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767601.333644:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767601/real 1775767601] req@ffff9d8188742300 x1862023066558080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767601.333656:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066558080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767601.333665:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767601.333668:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767601.333672:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767601.333677:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767601.333698:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767601.333708:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767601/real 1775767601] req@ffff9d8188742680 x1862023066558208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767601.333720:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066558208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767601.333729:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767601.333732:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767601.333736:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767601.973117:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775767602.357037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767602.997108:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767603.381057:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767604.021130:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767604.405043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767605.045080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767605.429076:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767606.069131:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767606.453006:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767606.453017:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767606.453023:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775767606.453025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767606.453035:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4437 00000100:00080000:3.0:1775767606.453046:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767606.453051:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767606.453103:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066558336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767606.453132:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767606.453136:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767606.453138:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767606.453142:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4437 00000100:00080000:3.0:1775767606.453146:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767606.453150:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767606.453160:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767606.453176:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066558464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767606.453190:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767606.453193:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066558336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767606.453195:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767606.453199:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767606.453207:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4437 00000100:00080000:3.0:1775767606.453210:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767606.453213:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767606.453218:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767606.453228:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066558592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767606.453236:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767606.453238:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767606.453240:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767606.453242:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4437 00000100:00080000:3.0:1775767606.453252:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767606.453254:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767606.453258:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767606.453260:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767606.453270:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066558720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767606.453278:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767606.453280:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767606.453282:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767606.453284:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4437 00000100:00080000:3.0:1775767606.453287:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767606.453289:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767606.453293:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767606.453304:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066558848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767606.453312:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767606/real 1775767606] req@ffff9d8188741880 x1862023066558336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767606.453329:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066558336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767606.453338:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767606.453341:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767606.453344:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066558336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767606.453363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767606.453382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767606.453391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767606/real 1775767606] req@ffff9d8188743800 x1862023066558464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767606.453399:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066558464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767606.453404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767606.453407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767606.453409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767606.453413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767606.453423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767606.453429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767606/real 1775767606] req@ffff9d8188741f80 x1862023066558592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767606.453437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066558592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767606.453442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767606.453444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767606.453446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767606.453449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767606.453459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767606.453464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767606/real 1775767606] req@ffff9d8188740000 x1862023066558720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767606.453476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066558720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767606.453482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767606.453484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767606.453486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767606.453489:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767606.453498:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767606.453503:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767606/real 1775767606] req@ffff9d8188740700 x1862023066558848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767606.453510:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066558848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767606.453524:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767606.453526:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767606.453531:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066558848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767607.093031:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767607.476941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767608.117126:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767608.500956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767609.141097:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767609.524962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767610.165099:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767610.549024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767611.189052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767611.572989:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767611.572998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767611.572998:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767611.573002:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767611.573010:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4443 00000100:00080000:3.0:1775767611.573020:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767611.573025:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767611.573058:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066558976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767611.573090:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767611.573093:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767611.573095:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767611.573097:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4443 00000100:00080000:3.0:1775767611.573100:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767611.573102:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767611.573109:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767611.573117:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066559104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767611.573125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767611.573128:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767611.573131:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767611.573136:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4443 00000100:00080000:3.0:1775767611.573140:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767611.573144:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066558976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767611.573147:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767611.573155:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767611.573169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066559232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767611.573181:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767611.573185:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767611.573187:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767611.573191:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4443 00000100:00080000:3.0:1775767611.573196:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767611.573199:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767611.573205:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767611.573208:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767611.573224:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066559360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767611.573234:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767611.573238:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767611.573240:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767611.573245:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4443 00000100:00080000:3.0:1775767611.573249:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767611.573254:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767611.573260:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767611/real 1775767611] req@ffff9d8188740380 x1862023066558976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767611.573260:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767611.573277:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066559488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767611.573287:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066558976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767611.573301:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767611.573306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767611.573310:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066558976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767611.573318:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767611.573342:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767611.573356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767611/real 1775767611] req@ffff9d8188741c00 x1862023066559104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767611.573370:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066559104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767611.573380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767611.573384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767611.573388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767611.573410:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767611.573430:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767611.573439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767611/real 1775767611] req@ffff9d8188742a00 x1862023066559232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767611.573447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066559232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767611.573453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767611.573455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767611.573457:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767611.573461:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767611.573473:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767611.573478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767611/real 1775767611] req@ffff9d8188740700 x1862023066559360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767611.573486:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066559360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767611.573492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767611.573494:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767611.573496:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767611.573499:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767611.573509:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767611.573514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767611/real 1775767611] req@ffff9d8188740000 x1862023066559488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767611.573521:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066559488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767611.573526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767611.573528:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767611.573530:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767612.213070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767612.597004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767613.237048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767613.620991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767614.261098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767614.644996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767615.285139:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767615.669046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767616.309151:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767616.692962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767616.693018:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767616.693029:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767616.693034:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767616.693045:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4448 00000100:00080000:3.0:1775767616.693057:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767616.693065:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767616.693104:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066559616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767616.693135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767616.693139:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767616.693141:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767616.693146:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4448 00000100:00080000:3.0:1775767616.693150:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767616.693154:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767616.693163:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767616.693179:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066559744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767616.693194:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066559616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767616.693194:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767616.693202:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767616.693205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767616.693211:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4448 00000100:00080000:3.0:1775767616.693215:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767616.693219:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767616.693225:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767616.693239:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066559872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767616.693260:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767616.693264:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767616.693266:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767616.693270:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4448 00000800:00000100:0.0:1775767616.693275:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767616.693275:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767616.693282:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767616.693288:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767616.693307:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066560000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767616.693317:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767616.693321:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767616.693323:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767616.693329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767616/real 1775767616] req@ffff9d8188741f80 x1862023066559616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767616.693330:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4448 00000100:00080000:3.0:1775767616.693336:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767616.693343:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767616.693351:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767616.693362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066559616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767616.693366:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066560128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767616.693390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767616.693397:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767616.693403:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066559616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767616.693429:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767616.693453:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767616.693467:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767616/real 1775767616] req@ffff9d8188743800 x1862023066559744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767616.693480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066559744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767616.693489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767616.693492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767616.693495:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767616.693502:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767616.693521:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767616.693530:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767616/real 1775767616] req@ffff9d8188741880 x1862023066559872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767616.693548:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066559872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767616.693556:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767616.693560:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767616.693564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066559872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767616.693570:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767616.693591:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767616.693602:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767616/real 1775767616] req@ffff9d8188742680 x1862023066560000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767616.693615:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066560000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767616.693624:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767616.693627:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767616.693630:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767616.693636:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767616.693655:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767616.693666:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767616/real 1775767616] req@ffff9d8188742300 x1862023066560128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767616.693677:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066560128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767616.693692:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767616.693695:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767616.693699:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767617.333108:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767617.717017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767618.357089:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767618.740945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767619.381089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767619.765014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767620.405057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767620.788944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767621.429024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767621.812988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767621.812991:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767621.813000:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767621.813004:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767621.813014:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4453 00000100:00080000:3.0:1775767621.813025:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767621.813034:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767621.813073:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066560256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767621.813098:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767621.813101:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767621.813102:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767621.813105:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4453 00000100:00080000:3.0:1775767621.813108:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767621.813111:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767621.813119:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767621.813129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066560384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767621.813136:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767621.813139:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767621.813140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767621.813143:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4453 00000100:00080000:3.0:1775767621.813144:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767621.813147:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767621.813150:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767621.813156:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066560512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767621.813163:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767621.813166:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775767621.813167:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066560256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767621.813168:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767621.813172:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4453 00000100:00080000:3.0:1775767621.813176:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767621.813179:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767621.813183:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767621.813192:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066560640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767621.813198:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767621.813200:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767621.813202:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767621.813204:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4453 00000100:00080000:3.0:1775767621.813207:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767621.813209:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767621.813212:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767621.813218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066560768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767621.813237:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767621.813283:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767621/real 1775767621] req@ffff9d8188740e00 x1862023066560256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767621.813303:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066560256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767621.813315:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767621.813319:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767621.813324:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066560256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767621.813332:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767621.813356:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767621.813368:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767621/real 1775767621] req@ffff9d8188741180 x1862023066560384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767621.813380:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066560384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767621.813390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767621.813403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767621.813409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767621.813433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767621.813454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767621.813466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767621/real 1775767621] req@ffff9d8188743100 x1862023066560512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767621.813480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066560512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767621.813489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767621.813493:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767621.813496:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767621.813503:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767621.813522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767621.813533:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767621/real 1775767621] req@ffff9d8188743480 x1862023066560640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767621.813545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066560640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767621.813556:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767621.813559:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767621.813562:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767621.813568:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767621.813589:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767621.813599:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767621/real 1775767621] req@ffff9d8188743b80 x1862023066560768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767621.813612:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066560768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767621.813621:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767621.813625:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767621.813628:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066560768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767622.453039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767622.836973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767623.477028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767623.860978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767624.501069:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767624.884978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767625.525046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767625.908962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767626.549044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767626.932964:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767626.932965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767626.932974:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767626.932979:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767626.932989:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4458 00000100:00080000:3.0:1775767626.932999:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767626.933006:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767626.933046:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066560896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767626.933069:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767626.933071:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767626.933073:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767626.933075:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4458 00000100:00080000:3.0:1775767626.933078:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767626.933081:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767626.933088:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767626.933098:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066561024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767626.933109:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767626.933112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767626.933114:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767626.933118:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4458 00000100:00100000:0.0:1775767626.933124:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066560896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767626.933124:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767626.933133:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767626.933139:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767626.933152:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066561152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767626.933164:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767626.933168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767626.933170:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767626.933174:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4458 00000100:00080000:3.0:1775767626.933191:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767626.933194:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767626.933199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767626.933200:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767626.933215:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066561280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767626.933222:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767626.933225:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767626.933227:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767626.933233:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4458 00000100:00080000:3.0:1775767626.933238:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767626.933241:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767626/real 1775767626] req@ffff9d8188742d80 x1862023066560896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767626.933246:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767626.933256:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767626.933262:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066560896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767626.933271:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767626.933273:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066561408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767626.933277:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767626.933283:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066560896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767626.933296:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767626.933313:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767626.933321:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767626/real 1775767626] req@ffff9d8188740a80 x1862023066561024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767626.933333:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066561024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767626.933338:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767626.933340:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767626.933342:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767626.933359:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767626.933370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767626.933376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767626/real 1775767626] req@ffff9d8188741500 x1862023066561152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767626.933384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066561152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767626.933389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767626.933391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767626.933393:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767626.933396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767626.933407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767626.933412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767626/real 1775767626] req@ffff9d8188743b80 x1862023066561280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767626.933419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066561280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767626.933429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767626.933431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767626.933434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767626.933437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767626.933446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767626.933451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767626/real 1775767626] req@ffff9d8188743480 x1862023066561408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767626.933457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066561408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767626.933462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767626.933465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767626.933467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767627.573077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767627.956993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767628.597063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767628.980942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767629.621090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767630.004975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767630.645052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767631.028966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767631.669095:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767632.052979:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767632.052983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767632.052990:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767632.052994:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767632.053002:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4463 00000100:00080000:3.0:1775767632.053011:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767632.053017:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767632.053050:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066561536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767632.053075:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767632.053077:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767632.053078:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767632.053081:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4463 00000100:00080000:3.0:1775767632.053083:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767632.053086:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767632.053093:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767632.053101:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066561664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767632.053109:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767632.053111:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767632.053113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767632.053116:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4463 00000100:00080000:3.0:1775767632.053118:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767632.053120:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767632.053123:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767632.053130:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066561792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767632.053137:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767632.053140:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767632.053142:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767632.053149:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066561536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767632.053149:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4463 00000100:00080000:3.0:1775767632.053160:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767632.053165:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767632.053169:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767632.053181:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066561920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767632.053187:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767632.053189:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767632.053191:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767632.053194:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4463 00000100:00080000:3.0:1775767632.053196:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767632.053199:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767632.053203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767632.053210:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066562048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767632.053245:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767632.053297:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767632/real 1775767632] req@ffff9d8188743100 x1862023066561536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767632.053314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066561536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767632.053323:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767632.053327:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767632.053331:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066561536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767632.053338:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767632.053356:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767632.053372:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767632/real 1775767632] req@ffff9d8188741180 x1862023066561664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767632.053381:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066561664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767632.053387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767632.053390:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767632.053392:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767632.053413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767632.053425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767632.053431:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767632/real 1775767632] req@ffff9d8188740e00 x1862023066561792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767632.053438:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066561792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767632.053450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767632.053454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767632.053458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767632.053465:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767632.053487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767632.053499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767632/real 1775767632] req@ffff9d8188742300 x1862023066561920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767632.053513:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066561920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767632.053529:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767632.053532:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767632.053535:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066561920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767632.053539:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767632.053556:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767632.053563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767632/real 1775767632] req@ffff9d8188742680 x1862023066562048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767632.053570:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066562048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767632.053577:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767632.053579:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767632.053581:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767632.693010:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767633.076946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767633.717108:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767634.100986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767634.741056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767635.125005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767635.765111:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767636.149001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767636.789033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767637.173004:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767637.173009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767637.173018:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767637.173023:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767637.173033:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4468 00000100:00080000:3.0:1775767637.173045:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767637.173053:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767637.173102:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066562176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767637.173135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767637.173138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767637.173140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767637.173145:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4468 00000100:00080000:3.0:1775767637.173149:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767637.173154:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767637.173167:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767637.173180:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066562176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767637.173186:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066562304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767637.173198:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767637.173202:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767637.173204:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767637.173209:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4468 00000100:00080000:3.0:1775767637.173213:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767637.173218:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767637.173223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767637.173234:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066562432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767637.173244:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767637.173248:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767637.173250:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767637.173254:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767637.173258:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4468 00000100:00080000:3.0:1775767637.173263:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767637.173267:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767637.173274:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767637.173288:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066562560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767637.173299:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767637.173304:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767637/real 1775767637] req@ffff9d8188741880 x1862023066562176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767637.173305:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767637.173310:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767637.173319:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4468 00000100:00080000:3.0:1775767637.173326:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767637.173330:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767637.173336:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767637.173340:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066562176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767637.173349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767637.173352:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066562688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767637.173355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767637.173362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066562176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767637.173389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767637.173408:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767637.173417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767637/real 1775767637] req@ffff9d8188743800 x1862023066562304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767637.173426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066562304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767637.173432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767637.173434:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767637.173436:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767637.173440:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767637.173451:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767637.173456:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767637/real 1775767637] req@ffff9d8188741f80 x1862023066562432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767637.173464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066562432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767637.173474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767637.173476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767637.173479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767637.173482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767637.173493:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767637.173498:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767637/real 1775767637] req@ffff9d8188740000 x1862023066562560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767637.173506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066562560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767637.173512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767637.173514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767637.173516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767637.173519:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767637.173535:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767637.173540:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767637/real 1775767637] req@ffff9d8188740700 x1862023066562688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767637.173548:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066562688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767637.173553:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767637.173555:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767637.173558:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767637.813123:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767638.196952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767638.837069:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767639.221012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767639.861046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767640.244941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767640.885040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767641.268981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767641.909071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767642.292978:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767642.292981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767642.292990:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767642.292994:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767642.293005:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4473 00000100:00080000:3.0:1775767642.293015:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767642.293021:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767642.293054:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066562816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767642.293079:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767642.293083:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767642.293084:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767642.293088:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4473 00000100:00080000:3.0:1775767642.293092:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767642.293096:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767642.293105:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767642.293119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066562944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767642.293129:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767642.293131:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066562816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767642.293132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767642.293134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767642.293138:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4473 00000100:00080000:3.0:1775767642.293140:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767642.293143:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767642.293148:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767642.293158:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066563072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767642.293166:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767642.293169:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767642.293171:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767642.293174:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4473 00000100:00080000:3.0:1775767642.293186:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767642.293190:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767642.293195:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767642.293204:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767642.293206:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066563200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767642.293214:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767642.293217:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767642.293218:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767642.293222:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4473 00000100:00080000:3.0:1775767642.293224:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767642.293228:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767642.293232:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767642.293241:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066563328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767642.293248:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767642/real 1775767642] req@ffff9d8188742a00 x1862023066562816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767642.293265:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066562816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767642.293276:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767642.293280:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767642.293284:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066562816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767642.293304:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767642.293321:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767642.293330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767642/real 1775767642] req@ffff9d8188741c00 x1862023066562944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767642.293339:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066562944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767642.293345:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767642.293347:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767642.293349:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066562944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767642.293352:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767642.293363:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767642.293369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767642/real 1775767642] req@ffff9d8188740380 x1862023066563072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767642.293376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066563072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767642.293381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767642.293383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767642.293384:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767642.293388:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767642.293397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767642.293402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767642/real 1775767642] req@ffff9d8188740700 x1862023066563200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767642.293409:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066563200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767642.293416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767642.293417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767642.293419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767642.293422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767642.293431:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767642.293436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767642/real 1775767642] req@ffff9d8188740000 x1862023066563328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767642.293448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066563328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767642.293453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767642.293454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767642.293456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767642.933057:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767643.317013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767643.957034:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767644.341007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767644.981031:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767645.364972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767646.005107:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767646.388988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767647.029040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767647.412946:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767647.412955:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767647.412959:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767647.412967:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4478 00000100:00100000:3.0:1775767647.412967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767647.412978:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767647.412985:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767647.413017:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066563456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767647.413046:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767647.413050:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767647.413051:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767647.413054:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4478 00000100:00080000:2.0:1775767647.413056:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767647.413058:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767647.413065:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767647.413076:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066563584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767647.413084:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767647.413086:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767647.413087:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767647.413090:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4478 00000100:00080000:2.0:1775767647.413092:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767647.413097:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066563456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767647.413097:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767647.413102:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767647.413110:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066563712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767647.413117:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767647.413119:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767647.413120:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767647.413122:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4478 00000100:00080000:2.0:1775767647.413124:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767647.413126:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767647.413130:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767647.413139:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066563840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767647.413150:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767647.413153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767647.413154:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767647.413156:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4478 00000100:00080000:2.0:1775767647.413158:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767647.413162:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767647.413168:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767647.413168:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767647.413179:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066563968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767647.413221:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767647/real 1775767647] req@ffff9d81984a2300 x1862023066563456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767647.413241:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066563456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767647.413253:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767647.413257:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767647.413261:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066563456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767647.413270:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767647.413293:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767647.413305:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767647/real 1775767647] req@ffff9d81984a2a00 x1862023066563584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767647.413317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066563584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767647.413327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767647.413331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767647.413334:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767647.413358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767647.413378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767647.413388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767647/real 1775767647] req@ffff9d81984a0700 x1862023066563712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767647.413401:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066563712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767647.413410:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767647.413413:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767647.413417:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767647.413423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767647.413442:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767647.413453:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767647/real 1775767647] req@ffff9d81984a2d80 x1862023066563840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767647.413465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066563840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767647.413475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767647.413479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767647.413482:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767647.413489:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767647.413509:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767647.413520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767647/real 1775767647] req@ffff9d81984a3480 x1862023066563968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767647.413533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066563968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767647.413542:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767647.413545:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767647.413548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066563968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767648.053052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767648.436962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767649.077050:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767649.460968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767650.101029:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767650.484950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767651.125097:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767651.508963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767652.149040:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767652.532947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767652.532956:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767652.532962:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767652.532965:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767652.532973:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4484 00000100:00080000:2.0:1775767652.532982:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767652.532988:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767652.533019:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066564096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767652.533041:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767652.533044:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767652.533045:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767652.533047:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4484 00000100:00080000:2.0:1775767652.533050:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767652.533052:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767652.533059:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767652.533066:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066564224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767652.533074:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767652.533076:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767652.533077:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767652.533080:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4484 00000100:00080000:2.0:1775767652.533082:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767652.533084:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767652.533088:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767652.533095:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066564096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767652.533096:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066564352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767652.533110:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767652.533112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767652.533113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767652.533115:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4484 00000100:00080000:2.0:1775767652.533117:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767652.533119:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767652.533122:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767652.533130:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066564480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767652.533140:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767652.533142:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767652.533144:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767652.533145:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4484 00000100:00080000:2.0:1775767652.533150:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767652.533152:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767652.533154:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767652.533161:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767652.533165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066564608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767652.533208:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767652/real 1775767652] req@ffff9d81984a3b80 x1862023066564096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767652.533229:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066564096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767652.533241:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767652.533246:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767652.533250:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066564096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767652.533258:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767652.533282:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767652.533292:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767652/real 1775767652] req@ffff9d81984a0000 x1862023066564224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767652.533308:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066564224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767652.533317:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767652.533320:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767652.533324:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767652.533344:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767652.533360:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767652.533370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767652/real 1775767652] req@ffff9d81984a2680 x1862023066564352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767652.533382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066564352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767652.533391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767652.533394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767652.533398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767652.533404:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767652.533420:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767652.533438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767652/real 1775767652] req@ffff9d81984a0e00 x1862023066564480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767652.533459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066564480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767652.533470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767652.533473:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767652.533477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767652.533483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767652.533501:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767652.533510:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767652/real 1775767652] req@ffff9d81984a1f80 x1862023066564608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767652.533523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066564608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767652.533532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767652.533565:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767652.533569:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767653.173058:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767653.557009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767654.197001:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767654.581010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767655.221041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767655.604953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767656.245064:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767656.628984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767657.269062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767657.653004:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767657.653008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767657.653017:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767657.653022:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767657.653035:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4489 00000100:00080000:1.0:1775767657.653054:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767657.653062:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767657.653109:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066564736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767657.653135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767657.653138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767657.653140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767657.653143:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4489 00000100:00080000:1.0:1775767657.653146:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767657.653149:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767657.653159:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767657.653169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066564864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767657.653177:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767657.653180:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767657.653182:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767657.653187:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066564736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767657.653187:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4489 00000100:00080000:1.0:1775767657.653191:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767657.653203:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767657.653207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767657.653218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066564992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767657.653225:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767657.653228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767657.653230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767657.653233:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4489 00000100:00080000:1.0:1775767657.653235:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767657.653238:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767657.653243:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767657.653250:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066565120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767657.653256:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767657.653259:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767657.653260:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767657.653263:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4489 00000100:00080000:1.0:1775767657.653266:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767657.653269:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767657.653272:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767657.653278:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066565248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767657.653334:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767657.653371:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767657.653414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767657.653442:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767658.293039:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767658.676980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767659.317046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767659.700943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767660.341026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767660.724976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775767660.725117:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775767660.725157:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775767660.725175:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767660.725216:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767660.725228:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767660.725240:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767660.725252:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767660.725272:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767657/real 1775767660] req@ffff9d8190553480 x1862023066564864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767660.725299:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066564864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767660.725312:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767660.725317:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767660.725321:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767660.725329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767657/real 1775767660] req@ffff9d8190553100 x1862023066564992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767660.725343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066564992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767660.725351:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767660.725355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767660.725359:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066564992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767660.725366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767657/real 1775767660] req@ffff9d8190551500 x1862023066565120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767660.725379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066565120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767660.725389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767660.725393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767660.725396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767660.725403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767657/real 1775767660] req@ffff9d8190551880 x1862023066565248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767660.725414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066565248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767660.725422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767660.725425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767660.725429:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565248:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767660.725435:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767657/real 1775767660] req@ffff9d8190550e00 x1862023066564736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767660.725447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066564736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767660.725490:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767660.725495:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767660.725499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066564736:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775767661.365041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767661.748985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767662.389022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767662.772992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767662.772994:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767662.773002:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767662.773005:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767662.773013:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4494 00000100:00080000:1.0:1775767662.773023:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767662.773029:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767662.773059:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066565376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767662.773082:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767662.773085:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767662.773086:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767662.773089:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4494 00000100:00080000:1.0:1775767662.773091:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767662.773093:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767662.773102:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767662.773110:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066565504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767662.773118:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767662.773121:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066565376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767662.773122:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767662.773125:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767662.773132:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4494 00000100:00080000:1.0:1775767662.773134:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767662.773136:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767662.773140:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767662.773148:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066565632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767662.773154:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767662.773157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767662.773158:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767662.773161:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4494 00000100:00080000:1.0:1775767662.773173:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767662.773175:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767662.773178:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767662.773185:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767662.773188:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066565760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767662.773195:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767662.773197:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767662.773199:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767662.773202:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4494 00000100:00080000:1.0:1775767662.773206:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767662.773208:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767662.773211:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767662.773218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066565888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767662.773240:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767662/real 1775767662] req@ffff9d8190550e00 x1862023066565376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767662.773260:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066565376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767662.773270:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767662.773275:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767662.773279:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066565376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767662.773307:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767662.773330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767662.773344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767662/real 1775767662] req@ffff9d8190551880 x1862023066565504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767662.773357:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066565504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767662.773367:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767662.773370:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767662.773374:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767662.773383:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767662.773403:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767662.773413:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767662/real 1775767662] req@ffff9d8190551500 x1862023066565632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767662.773424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066565632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767662.773432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767662.773450:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767662.773453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767662.773460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767662.773481:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767662.773492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767662/real 1775767662] req@ffff9d8190553100 x1862023066565760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767662.773505:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066565760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767662.773517:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767662.773521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767662.773524:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767662.773531:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767662.773551:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767662.773563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767662/real 1775767662] req@ffff9d8190553480 x1862023066565888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767662.773575:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066565888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767662.773599:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767662.773603:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767662.773606:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066565888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767663.413025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767663.796960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767664.437055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767664.821013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767665.461066:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767665.844973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767666.485039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767666.868975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767667.509041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767667.892951:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767667.892953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767667.892961:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767667.892965:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767667.892973:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4499 00000100:00080000:1.0:1775767667.892982:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767667.892988:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767667.893030:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066566016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767667.893052:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767667.893055:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767667.893056:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767667.893059:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4499 00000100:00080000:1.0:1775767667.893061:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767667.893064:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767667.893071:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767667.893081:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066566144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767667.893085:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066566016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767667.893100:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767667.893103:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767667.893104:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767667.893107:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4499 00000100:00080000:1.0:1775767667.893109:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767667.893111:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767667.893115:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767667.893124:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066566272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767667.893130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767667.893133:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767667.893134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767667.893136:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4499 00000100:00080000:1.0:1775767667.893145:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767667.893147:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767667.893150:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767667.893152:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767667.893160:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066566400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767667.893167:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767667.893170:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767667.893171:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767667.893173:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4499 00000100:00080000:1.0:1775767667.893176:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767667.893178:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767667.893181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767667.893189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066566528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767667.893200:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767667/real 1775767667] req@ffff9d8190553480 x1862023066566016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767667.893222:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066566016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767667.893231:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767667.893234:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767667.893237:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066566016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767667.893257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767667.893274:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767667.893282:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767667/real 1775767667] req@ffff9d8190553100 x1862023066566144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767667.893290:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066566144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767667.893296:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767667.893298:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767667.893300:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767667.893305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767667.893323:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767667.893334:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767667/real 1775767667] req@ffff9d8190551500 x1862023066566272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767667.893346:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066566272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767667.893355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767667.893359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767667.893362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767667.893368:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767667.893402:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767667.893414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767667/real 1775767667] req@ffff9d8190551880 x1862023066566400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767667.893426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066566400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767667.893437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767667.893441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767667.893444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767667.893449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767667.893465:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767667.893471:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767667/real 1775767667] req@ffff9d8190550e00 x1862023066566528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767667.893479:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066566528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767667.893484:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767667.893487:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767667.893489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767668.533095:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767668.917027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767669.557077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767669.941010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767670.581125:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767670.965009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767671.605064:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767671.988959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767672.629050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767673.012930:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767673.012938:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767673.012942:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767673.012951:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4504 00000100:00080000:1.0:1775767673.012961:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767673.012968:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767673.013002:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066566656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767673.013010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767673.013032:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767673.013036:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767673.013037:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767673.013043:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4504 00000100:00080000:1.0:1775767673.013046:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767673.013048:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767673.013056:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767673.013065:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066566784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767673.013073:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767673.013077:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767673.013078:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767673.013083:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4504 00000100:00080000:1.0:1775767673.013086:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767673.013089:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767673.013096:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767673.013097:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066566656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767673.013110:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066566912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767673.013119:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767673.013122:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767673.013123:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767673.013126:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4504 00000100:00080000:1.0:1775767673.013128:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767673.013130:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767673.013134:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767673.013144:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066567040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767673.013149:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767673.013151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767673.013152:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767673.013155:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4504 00000100:00080000:1.0:1775767673.013158:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767673.013161:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767673.013165:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767673.013172:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066567168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1775767673.013199:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767673.013261:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767673/real 1775767673] req@ffff9d8190550e00 x1862023066566656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767673.013280:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066566656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767673.013290:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767673.013293:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767673.013297:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066566656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767673.013303:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767673.013322:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767673.013331:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767673/real 1775767673] req@ffff9d8190551880 x1862023066566784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767673.013366:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066566784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767673.013372:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767673.013375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767673.013377:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767673.013402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767673.013419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767673.013426:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767673/real 1775767673] req@ffff9d8190551500 x1862023066566912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767673.013434:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066566912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767673.013440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767673.013442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767673.013444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066566912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767673.013448:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767673.013459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767673.013464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767673/real 1775767673] req@ffff9d8190553100 x1862023066567040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767673.013472:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066567040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767673.013478:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767673.013480:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767673.013482:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767673.013485:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767673.013495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767673.013500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767673/real 1775767673] req@ffff9d8190553480 x1862023066567168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767673.013512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066567168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767673.013518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767673.013520:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767673.013522:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767673.653024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767674.036967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767674.677069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767675.061013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767675.701094:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767676.085042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767676.725062:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767677.108995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767677.749033:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767678.132994:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767678.133002:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775767678.133006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767678.133006:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767678.133017:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4509 00000100:00080000:1.0:1775767678.133025:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767678.133032:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767678.133067:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066567296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767678.133092:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767678.133094:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767678.133096:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767678.133098:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4509 00000100:00080000:1.0:1775767678.133100:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767678.133103:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767678.133109:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767678.133117:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066567424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767678.133124:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767678.133131:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767678.133132:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767678.133135:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4509 00000100:00080000:1.0:1775767678.133137:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767678.133139:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767678.133142:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767678.133149:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066567552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767678.133161:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767678.133164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767678.133165:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767678.133168:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4509 00000100:00080000:1.0:1775767678.133170:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767678.133176:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767678.133177:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066567296:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775767678.133181:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767678.133195:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066567680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767678.133206:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767678.133210:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767678.133212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767678.133216:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4509 00000100:00080000:1.0:1775767678.133236:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767678.133240:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767678.133244:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775767678.133246:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767678.133256:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066567808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767678.133306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767678/real 1775767678] req@ffff9d8190553480 x1862023066567296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767678.133322:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066567296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767678.133330:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767678.133333:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767678.133336:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066567296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767678.133340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767678.133357:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767678.133365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767678/real 1775767678] req@ffff9d8190553100 x1862023066567424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767678.133378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066567424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767678.133383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767678.133385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767678.133386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767678.133390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767678.133400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767678.133405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767678/real 1775767678] req@ffff9d8190551500 x1862023066567552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767678.133412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066567552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767678.133417:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767678.133419:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767678.133420:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767678.133437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767678.133448:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767678.133454:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767678/real 1775767678] req@ffff9d8190551880 x1862023066567680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767678.133461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066567680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767678.133471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767678.133473:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767678.133475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767678.133478:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767678.133487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767678.133492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767678/real 1775767678] req@ffff9d8190550e00 x1862023066567808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767678.133498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066567808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767678.133504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767678.133505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767678.133507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066567808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767678.773090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767679.156947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767679.797000:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767680.180968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767680.821047:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767681.204973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767681.845056:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767682.228998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767682.869028:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767683.252981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767683.252988:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767683.252995:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767683.253000:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767683.253010:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4514 00000100:00080000:1.0:1775767683.253021:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767683.253029:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767683.253067:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066567936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767683.253094:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767683.253099:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767683.253101:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767683.253104:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4514 00000100:00080000:1.0:1775767683.253108:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767683.253120:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767683.253130:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767683.253145:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066568064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767683.253157:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767683.253161:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767683.253164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767683.253164:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066567936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767683.253175:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4514 00000100:00080000:1.0:1775767683.253178:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767683.253182:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767683.253188:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767683.253209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066568192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767683.253221:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1775767683.253224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767683.253228:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767683.253231:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767683.253250:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4514 00000100:00080000:1.0:1775767683.253256:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767683.253261:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767683.253268:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767683.253282:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066568320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767683.253292:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767683.253296:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767683.253299:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767683.253301:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767683/real 1775767683] req@ffff9d8190550e00 x1862023066567936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767683.253306:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4514 00000100:00080000:1.0:1775767683.253313:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767683.253321:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767683.253327:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767683.253334:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066567936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767683.253348:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767683.253354:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066568448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767683.253354:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767683.253359:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066567936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767683.253384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767683.253404:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767683.253412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767683/real 1775767683] req@ffff9d8190551880 x1862023066568064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767683.253420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066568064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767683.253425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767683.253428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767683.253430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767683.253433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767683.253443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767683.253448:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767683/real 1775767683] req@ffff9d8190551500 x1862023066568192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767683.253455:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066568192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767683.253460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767683.253462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767683.253465:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767683.253468:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767683.253479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767683.253484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767683/real 1775767683] req@ffff9d8190553100 x1862023066568320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767683.253490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066568320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767683.253496:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767683.253498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767683.253500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767683.253503:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767683.253512:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767683.253517:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767683/real 1775767683] req@ffff9d8190553480 x1862023066568448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767683.253524:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066568448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767683.253530:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767683.253531:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767683.253533:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767683.893098:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767684.277031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767684.917084:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767685.300965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767685.941072:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767686.324966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767686.965085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767687.348954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767687.989059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767688.372962:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767688.372967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767688.372976:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767688.372981:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767688.372988:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4519 00000100:00080000:1.0:1775767688.373008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767688.373013:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767688.373048:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066568576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767688.373071:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767688.373074:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767688.373075:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767688.373088:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4519 00000100:00080000:1.0:1775767688.373091:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767688.373093:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767688.373100:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767688.373107:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066568704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767688.373114:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767688.373117:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767688.373118:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767688.373121:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4519 00000100:00080000:1.0:1775767688.373123:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767688.373125:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767688.373128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767688.373140:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066568832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767688.373144:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066568576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767688.373159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767688.373161:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767688.373162:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767688.373165:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4519 00000100:00080000:1.0:1775767688.373168:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767688.373170:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767688.373174:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767688.373185:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066568960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767688.373231:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767688.373235:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767688.373237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767688.373241:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4519 00000100:00080000:1.0:1775767688.373246:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767688.373250:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767688.373254:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767688.373261:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767688.373274:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066569088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767688.373308:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767688/real 1775767688] req@ffff9d8190553480 x1862023066568576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767688.373329:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066568576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767688.373340:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767688.373344:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767688.373348:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066568576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767688.373356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767688.373379:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767688.373391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767688/real 1775767688] req@ffff9d8190553100 x1862023066568704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767688.373404:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066568704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767688.373412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767688.373415:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767688.373419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767688.373440:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767688.373457:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767688.373466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767688/real 1775767688] req@ffff9d8190551500 x1862023066568832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767688.373479:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066568832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767688.373488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767688.373491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767688.373494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767688.373500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767688.373528:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767688.373538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767688/real 1775767688] req@ffff9d8190551880 x1862023066568960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767688.373551:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066568960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767688.373560:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767688.373563:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767688.373567:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066568960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767688.373572:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767688.373589:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767688.373597:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767688/real 1775767688] req@ffff9d8190550e00 x1862023066569088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767688.373609:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066569088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767688.373623:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767688.373626:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767688.373629:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767689.013033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767689.396961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767690.037082:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767690.420992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767691.061027:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767691.445000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767692.085055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767692.468975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767693.109048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767693.492989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767693.493011:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767693.493017:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767693.493021:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767693.493030:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4525 00000100:00080000:2.0:1775767693.493040:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767693.493047:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767693.493102:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066569216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767693.493163:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767693.493168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767693.493177:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767693.493184:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4525 00000100:00100000:3.0:1775767693.493185:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066569216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767693.493188:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767693.493194:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767693.493203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767693.493217:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066569344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767693.493225:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767693.493228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767693.493229:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767693.493231:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4525 00000100:00080000:2.0:1775767693.493241:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767693.493244:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767693.493247:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767693.493251:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767693.493258:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066569472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767693.493266:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767693.493268:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767693.493269:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767693.493272:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4525 00000100:00080000:2.0:1775767693.493275:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767693.493278:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767693.493282:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767693.493295:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767693/real 1775767693] req@ffff9d81984a1180 x1862023066569216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767693.493297:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066569600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767693.493321:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767693.493327:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767693.493329:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767693.493332:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066569216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767693.493336:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4525 00000100:00080000:2.0:1775767693.493343:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767693.493346:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767693.493348:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775767693.493352:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:2.0:1775767693.493354:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767693.493357:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066569216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767693.493367:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066569728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767693.493380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767693.493403:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767693.493411:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767693/real 1775767693] req@ffff9d81984a1880 x1862023066569344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767693.493420:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066569344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767693.493429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767693.493432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767693.493434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767693.493437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767693.493454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767693.493464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767693/real 1775767693] req@ffff9d81984a3800 x1862023066569472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767693.493477:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066569472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767693.493485:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767693.493489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767693.493493:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767693.493499:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767693.493518:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767693.493526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767693/real 1775767693] req@ffff9d81984a3100 x1862023066569600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767693.493534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066569600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767693.493541:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767693.493543:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767693.493545:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767693.493549:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767693.493560:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767693.493565:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767693/real 1775767693] req@ffff9d81984a1500 x1862023066569728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767693.493572:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066569728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767693.493577:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767693.493579:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767693.493581:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767694.133042:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767694.516974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767695.157061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767695.541003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767696.181062:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767696.564991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767697.205075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767697.589016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767698.229066:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767698.613008:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767698.613016:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767698.613020:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767698.613021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767698.613030:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4530 00000100:00080000:1.0:1775767698.613040:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767698.613046:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767698.613083:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066569856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767698.613128:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767698.613132:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767698.613134:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767698.613139:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4530 00000100:00080000:1.0:1775767698.613142:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767698.613147:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767698.613157:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767698.613168:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066569984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767698.613173:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066569856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767698.613182:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767698.613185:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767698.613186:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767698.613190:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4530 00000100:00080000:1.0:1775767698.613193:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767698.613195:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767698.613199:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767698.613216:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066570112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767698.613222:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767698.613225:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767698.613226:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767698.613230:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4530 00000800:00000100:3.0:1775767698.613234:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767698.613235:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767698.613240:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767698.613244:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767698.613252:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066570240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767698.613258:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767698.613260:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767698.613261:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767698.613264:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4530 00000100:00080000:1.0:1775767698.613266:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767698.613269:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767698.613272:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767698.613278:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767698/real 1775767698] req@ffff9d8190550e00 x1862023066569856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767698.613281:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066570368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767698.613312:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066569856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767698.613321:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767698.613334:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767698.613339:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066569856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767698.613358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767698.613374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767698.613383:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767698/real 1775767698] req@ffff9d8190551880 x1862023066569984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767698.613391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066569984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767698.613397:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767698.613399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767698.613401:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066569984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767698.613405:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767698.613416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767698.613421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767698/real 1775767698] req@ffff9d8190551500 x1862023066570112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767698.613428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066570112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767698.613433:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767698.613435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767698.613437:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767698.613441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767698.613452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767698.613457:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767698/real 1775767698] req@ffff9d8190553100 x1862023066570240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767698.613464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066570240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767698.613471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767698.613472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767698.613474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767698.613477:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767698.613487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767698.613492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767698/real 1775767698] req@ffff9d8190553480 x1862023066570368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767698.613499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066570368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767698.613505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767698.613506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767698.613508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767699.253094:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767699.637008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767700.277087:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767700.660969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767701.301080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767701.685030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767702.325102:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767702.708997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767703.349000:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767703.733031:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767703.733032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767703.733057:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767703.733063:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767703.733073:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4535 00000100:00080000:1.0:1775767703.733086:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767703.733094:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767703.733131:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066570496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767703.733153:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767703.733156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767703.733158:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767703.733160:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4535 00000100:00080000:1.0:1775767703.733163:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767703.733166:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767703.733173:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767703.733182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066570624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767703.733189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767703.733191:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767703.733193:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767703.733195:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4535 00000100:00080000:1.0:1775767703.733198:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767703.733200:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767703.733203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767703.733209:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066570496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767703.733211:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066570752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767703.733223:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767703.733225:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767703.733226:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767703.733229:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4535 00000100:00080000:1.0:1775767703.733232:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767703.733234:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767703.733237:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767703.733253:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066570880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767703.733260:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767703.733262:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767703.733263:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767703.733265:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4535 00000100:00080000:1.0:1775767703.733267:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1775767703.733271:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767703.733272:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767703.733279:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767703.733292:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066571008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767703.733315:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767703/real 1775767703] req@ffff9d8190553480 x1862023066570496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767703.733338:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066570496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767703.733349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767703.733354:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767703.733358:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066570496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767703.733366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767703.733391:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767703.733403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767703/real 1775767703] req@ffff9d8190553100 x1862023066570624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767703.733417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066570624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767703.733425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767703.733428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767703.733432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767703.733456:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767703.733477:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767703.733489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767703/real 1775767703] req@ffff9d8190551500 x1862023066570752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767703.733501:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066570752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767703.733515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767703.733519:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767703.733522:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767703.733529:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767703.733549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767703.733559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767703/real 1775767703] req@ffff9d8190551880 x1862023066570880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767703.733570:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066570880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767703.733577:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767703.733579:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767703.733581:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066570880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767703.733585:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767703.733598:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767703.733604:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767703/real 1775767703] req@ffff9d8190550e00 x1862023066571008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767703.733611:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066571008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767703.733616:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767703.733618:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767703.733620:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767704.373066:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767704.756963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767705.397097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767705.781068:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767706.421085:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767706.804975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767707.445101:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767707.829043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767708.469057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767708.852984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767708.853017:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767708.853025:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767708.853030:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767708.853040:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4540 00000100:00080000:2.0:1775767708.853051:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767708.853058:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767708.853095:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066571136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767708.853125:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767708.853132:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767708.853134:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767708.853137:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4540 00000100:00080000:2.0:1775767708.853141:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767708.853145:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767708.853153:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767708.853166:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066571264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767708.853171:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066571136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767708.853181:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767708.853184:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767708.853185:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767708.853190:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4540 00000100:00080000:2.0:1775767708.853193:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767708.853195:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767708.853198:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767708.853207:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066571392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767708.853213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767708.853216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767708.853218:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767708.853220:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4540 00000100:00080000:2.0:1775767708.853223:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767708.853225:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775767708.853230:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767708.853232:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767708.853243:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066571520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767708.853250:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767708.853252:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767708.853253:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767708.853255:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4540 00000100:00080000:2.0:1775767708.853258:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767708.853261:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767708.853264:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767708.853273:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767708/real 1775767708] req@ffff9d81984a0a80 x1862023066571136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767708.853275:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066571648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767708.853316:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066571136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767708.853339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767708.853349:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767708.853353:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066571136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767708.853374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767708.853393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767708.853402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767708/real 1775767708] req@ffff9d81984a1500 x1862023066571264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767708.853411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066571264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767708.853417:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767708.853419:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767708.853421:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767708.853427:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767708.853439:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767708.853445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767708/real 1775767708] req@ffff9d81984a3100 x1862023066571392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767708.853452:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066571392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767708.853457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767708.853459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767708.853461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767708.853464:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767708.853475:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767708.853480:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767708/real 1775767708] req@ffff9d81984a3800 x1862023066571520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767708.853487:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066571520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767708.853494:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767708.853496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767708.853497:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767708.853501:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767708.853510:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767708.853515:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767708/real 1775767708] req@ffff9d81984a1880 x1862023066571648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767708.853523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066571648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767708.853528:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767708.853530:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767708.853531:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767709.493120:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767709.877037:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767710.517073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767710.901029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767711.541092:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767711.925010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767712.565019:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767712.949006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767713.589120:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767713.972985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767713.972985:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767713.973000:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767713.973004:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767713.973016:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4545 00000100:00080000:2.0:1775767713.973026:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767713.973032:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767713.973067:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066571776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767713.973109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767713.973114:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767713.973116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767713.973120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4545 00000100:00080000:2.0:1775767713.973125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767713.973129:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767713.973138:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767713.973149:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066571904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767713.973161:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767713.973163:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066571776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767713.973168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767713.973178:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767713.973183:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4545 00000100:00080000:2.0:1775767713.973186:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767713.973188:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767713.973191:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767713.973208:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066572032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767713.973214:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767713.973216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767713.973218:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767713.973221:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4545 00000100:00080000:2.0:1775767713.973223:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767713.973225:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767713.973228:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767713.973235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066572160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767713.973247:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767713.973249:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767713.973250:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767713.973252:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4545 00000100:00080000:2.0:1775767713.973255:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767713.973257:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775767713.973269:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767713.973270:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767713.973281:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066572288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767713.973348:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767713/real 1775767713] req@ffff9d81984a1180 x1862023066571776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767713.973369:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066571776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767713.973379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767713.973383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767713.973386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066571776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767713.973412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767713.973435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767713.973445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767713/real 1775767713] req@ffff9d81984a1f80 x1862023066571904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767713.973459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066571904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767713.973466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767713.973468:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767713.973470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066571904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767713.973475:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767713.973487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767713.973493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767713/real 1775767713] req@ffff9d81984a0e00 x1862023066572032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767713.973502:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066572032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767713.973508:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767713.973510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767713.973512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767713.973516:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767713.973527:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767713.973533:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767713/real 1775767713] req@ffff9d81984a2680 x1862023066572160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767713.973541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066572160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767713.973552:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767713.973555:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767713.973557:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767713.973560:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767713.973571:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767713.973576:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767713/real 1775767713] req@ffff9d81984a0000 x1862023066572288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767713.973583:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066572288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767713.973589:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767713.973591:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767713.973593:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767714.613050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767714.997009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767715.637160:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767716.021016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767716.661107:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767717.045059:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767717.685080:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767718.069009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767718.709056:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767719.093008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767719.093009:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767719.093017:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767719.093022:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767719.093030:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4550 00000100:00080000:1.0:1775767719.093039:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767719.093045:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767719.093096:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066572416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767719.093123:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767719.093130:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767719.093132:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767719.093136:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4550 00000100:00080000:1.0:1775767719.093139:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767719.093144:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767719.093152:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767719.093164:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066572416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767719.093166:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066572544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767719.093180:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767719.093182:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767719.093183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767719.093186:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4550 00000100:00080000:1.0:1775767719.093189:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767719.093191:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767719.093194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767719.093202:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066572672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767719.093208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767719.093211:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767719.093212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767719.093214:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4550 00000100:00080000:1.0:1775767719.093227:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767719.093231:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:3.0:1775767719.093236:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775767719.093241:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767719.093258:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066572800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767719.093267:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767719.093271:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767719.093274:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767719.093278:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4550 00000100:00080000:1.0:1775767719.093283:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767719.093288:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767719.093291:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767719/real 1775767719] req@ffff9d8190550e00 x1862023066572416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767719.093295:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767719.093313:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066572928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767719.093321:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066572416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767719.093340:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767719.093345:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767719.093350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066572416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767719.093373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767719.093397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767719.093410:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767719/real 1775767719] req@ffff9d8190551880 x1862023066572544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767719.093423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066572544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767719.093432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767719.093435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767719.093439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767719.093445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767719.093465:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767719.093476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767719/real 1775767719] req@ffff9d8190551500 x1862023066572672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767719.093489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066572672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767719.093499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767719.093502:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767719.093506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767719.093513:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767719.093534:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767719.093545:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767719/real 1775767719] req@ffff9d8190553100 x1862023066572800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767719.093557:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066572800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767719.093567:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767719.093570:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767719.093574:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767719.093579:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767719.093598:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767719.093609:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767719/real 1775767719] req@ffff9d8190553480 x1862023066572928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767719.093622:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066572928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767719.093631:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767719.093634:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767719.093637:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066572928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767719.733066:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767720.116973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767720.757042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767721.140996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767721.781057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767722.164975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767722.805028:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767723.188979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767723.829070:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767724.212968:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767724.212969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767724.212978:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767724.212982:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767724.212990:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4555 00000100:00080000:1.0:1775767724.212998:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767724.213004:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767724.213039:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066573056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767724.213063:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767724.213070:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767724.213072:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767724.213074:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4555 00000100:00080000:1.0:1775767724.213077:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767724.213080:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767724.213088:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767724.213097:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066573184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767724.213104:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767724.213107:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767724.213109:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767724.213114:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4555 00000100:00100000:3.0:1775767724.213114:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066573056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767724.213119:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767724.213123:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767724.213128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767724.213135:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066573312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767724.213141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767724.213143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767724.213145:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767724.213148:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4555 00000100:00080000:1.0:1775767724.213150:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767724.213152:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767724.213155:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767724.213162:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066573440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767724.213172:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767724.213175:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767724.213176:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767724.213178:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4555 00000100:00080000:1.0:1775767724.213181:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767724.213183:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767724.213186:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767724.213194:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066573568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767724.213223:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767724.213248:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767724.213263:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767724.213276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767724.853068:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767725.237071:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767725.877024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767726.260968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767726.901087:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767727.285136:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767727.285246:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767727.285288:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767727.285305:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767727.285346:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767727.285355:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767727.285365:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767727.285376:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:3.0:1775767727.285393:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767724/real 1775767727] req@ffff9d8190553100 x1862023066573184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767727.285422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066573184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767727.285433:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767727.285438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767727.285443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767727.285452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767724/real 1775767727] req@ffff9d8190551500 x1862023066573312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767727.285464:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066573312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767727.285472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767727.285476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767727.285480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767727.285487:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767724/real 1775767727] req@ffff9d8190551880 x1862023066573440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767727.285499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066573440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767727.285509:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767727.285512:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767727.285516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767727.285522:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767724/real 1775767727] req@ffff9d8190550e00 x1862023066573568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767727.285532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066573568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767727.285539:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767727.285542:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767727.285544:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573568:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767727.285550:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767724/real 1775767727] req@ffff9d8190553480 x1862023066573056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767727.285561:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066573056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767727.285568:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767727.285572:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767727.285575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066573056:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775767727.925030:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767728.309049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767728.949020:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767729.332972:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767729.332983:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767729.332987:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767729.332995:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4560 00000100:00080000:1.0:1775767729.333005:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767729.333011:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767729.333019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775767729.333043:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066573696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767729.333068:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767729.333071:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767729.333072:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767729.333077:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4560 00000100:00080000:1.0:1775767729.333081:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767729.333085:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767729.333103:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767729.333119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066573824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767729.333122:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066573696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767729.333139:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767729.333143:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767729.333145:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767729.333151:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4560 00000100:00080000:1.0:1775767729.333155:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767729.333160:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767729.333165:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767729.333175:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066573952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767729.333184:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767729.333188:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767729.333190:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1775767729.333194:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767729.333199:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4560 00000100:00080000:1.0:1775767729.333206:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767729.333210:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767729.333216:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767729.333245:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767729/real 1775767729] req@ffff9d8190553480 x1862023066573696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767729.333246:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066574080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767729.333282:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767729.333286:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767729.333288:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1775767729.333290:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066573696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767729.333295:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4560 00000100:00080000:1.0:1775767729.333303:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767729.333308:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767729.333310:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:3.0:1775767729.333316:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:1.0:1775767729.333319:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767729.333322:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066573696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767729.333333:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066574208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767729.333348:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767729.333374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767729.333387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767729/real 1775767729] req@ffff9d8190550e00 x1862023066573824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767729.333399:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066573824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767729.333408:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767729.333411:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767729.333415:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767729.333423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767729.333442:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767729.333467:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767729/real 1775767729] req@ffff9d8190551880 x1862023066573952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767729.333480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066573952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767729.333488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767729.333491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767729.333494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066573952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767729.333500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767729.333519:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767729.333530:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767729/real 1775767729] req@ffff9d8190551500 x1862023066574080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767729.333541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066574080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767729.333550:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767729.333554:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767729.333556:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767729.333562:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767729.333580:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767729.333588:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767729/real 1775767729] req@ffff9d8190553100 x1862023066574208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767729.333600:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066574208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767729.333609:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767729.333612:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767729.333615:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767729.973032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767730.357092:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767730.997062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767731.380985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767732.021142:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767732.405003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767733.045074:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767733.428947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767734.069082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767734.452993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767734.452995:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767734.453002:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767734.453006:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767734.453014:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4565 00000100:00080000:1.0:1775767734.453024:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767734.453030:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767734.453072:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066574336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767734.453098:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767734.453102:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767734.453104:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767734.453107:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4565 00000100:00080000:1.0:1775767734.453110:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767734.453114:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767734.453122:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767734.453126:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066574336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767734.453137:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066574464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767734.453150:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767734.453153:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767734.453154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767734.453157:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4565 00000100:00080000:1.0:1775767734.453171:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767734.453173:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767734.453177:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1775767734.453181:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767734.453189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066574592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767734.453197:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767734.453199:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767734.453200:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767734.453203:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4565 00000100:00080000:1.0:1775767734.453205:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767734.453207:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767734.453212:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767734.453225:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066574720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767734.453225:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767734/real 1775767734] req@ffff9d8190553100 x1862023066574336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767734.453252:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767734.453256:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767734.453257:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767734.453260:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4565 00000100:00100000:3.0:1775767734.453263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066574336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767734.453264:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767734.453271:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767734.453279:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:3.0:1775767734.453281:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767734.453287:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775767734.453288:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066574848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767734.453293:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066574336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767734.453319:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767734.453340:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767734.453352:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767734/real 1775767734] req@ffff9d8190551500 x1862023066574464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767734.453374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066574464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767734.453382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767734.453386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767734.453389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767734.453396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767734.453416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767734.453424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767734/real 1775767734] req@ffff9d8190551880 x1862023066574592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767734.453432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066574592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767734.453438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767734.453440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767734.453442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767734.453445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767734.453457:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767734.453462:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767734/real 1775767734] req@ffff9d8190550e00 x1862023066574720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767734.453469:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066574720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767734.453480:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767734.453482:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767734.453484:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767734.453487:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767734.453496:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767734.453501:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767734/real 1775767734] req@ffff9d8190553480 x1862023066574848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767734.453508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066574848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767734.453513:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767734.453515:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767734.453517:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066574848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767735.093075:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767735.476974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767736.117017:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767736.501017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767737.141080:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767737.524943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767738.165045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767738.548968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767739.189039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767739.572951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767739.572963:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767739.572970:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767739.572975:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767739.572985:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4571 00000100:00080000:1.0:1775767739.572994:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767739.572999:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767739.573034:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066574976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767739.573060:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767739.573063:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767739.573064:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767739.573067:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4571 00000100:00080000:1.0:1775767739.573069:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767739.573072:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767739.573079:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767739.573087:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066575104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767739.573094:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767739.573097:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767739.573098:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767739.573101:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4571 00000100:00080000:1.0:1775767739.573103:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767739.573105:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767739.573108:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767739.573118:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066575232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767739.573120:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066574976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767739.573132:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767739.573134:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767739.573135:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767739.573138:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4571 00000100:00080000:1.0:1775767739.573140:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767739.573142:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767739.573145:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767739.573153:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066575360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767739.573164:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767739.573166:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767739.573167:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767739.573169:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4571 00000100:00080000:1.0:1775767739.573179:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767739.573181:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767739.573184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:3.0:1775767739.573187:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767739.573195:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066575488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767739.573233:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767739/real 1775767739] req@ffff9d8190553480 x1862023066574976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767739.573253:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066574976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767739.573264:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767739.573268:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767739.573272:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066574976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767739.573280:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767739.573300:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767739.573310:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767739/real 1775767739] req@ffff9d8190550e00 x1862023066575104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767739.573323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066575104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767739.573332:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767739.573335:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767739.573338:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767739.573359:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767739.573374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767739.573382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767739/real 1775767739] req@ffff9d8190551880 x1862023066575232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767739.573394:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066575232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767739.573410:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767739.573413:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767739.573416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767739.573422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767739.573437:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767739.573445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767739/real 1775767739] req@ffff9d8190551500 x1862023066575360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767739.573457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066575360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767739.573468:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767739.573470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767739.573474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767739.573479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767739.573494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767739.573502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767739/real 1775767739] req@ffff9d8190553100 x1862023066575488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767739.573514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066575488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767739.573523:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767739.573526:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767739.573529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767740.213023:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767740.597008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767741.237045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767741.621013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767742.261029:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767742.644992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767743.285067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767743.668979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767744.309058:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767744.693008:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767744.693020:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767744.693025:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767744.693034:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4576 00000100:00080000:1.0:1775767744.693046:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1775767744.693048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767744.693056:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767744.693097:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066575616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767744.693133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767744.693138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767744.693139:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767744.693144:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4576 00000100:00080000:1.0:1775767744.693148:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767744.693153:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767744.693162:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767744.693174:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767744.693179:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066575616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767744.693204:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767744.693206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767744.693208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767744.693213:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4576 00000100:00080000:1.0:1775767744.693215:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767744.693219:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767744.693223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767744.693231:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066575872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767744.693238:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767744.693241:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767744.693242:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767744.693245:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4576 00000100:00080000:1.0:1775767744.693260:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767744.693263:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767744.693266:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:3.0:1775767744.693272:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767744.693276:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066576000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767744.693283:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767744.693286:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767744.693288:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767744.693292:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4576 00000100:00080000:1.0:1775767744.693299:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767744.693302:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767744.693305:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767744.693311:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066576128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1775767744.693348:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767744/real 1775767744] req@ffff9d8190553100 x1862023066575616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767744.693370:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066575616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767744.693379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767744.693384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767744.693387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066575616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767744.693413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767744.693433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767744.693443:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767744/real 1775767744] req@ffff9d8190551500 x1862023066575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767744.693451:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066575744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767744.693458:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767744.693460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767744.693462:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767744.693466:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:3.0:1775767744.693477:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767744.693483:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767744/real 1775767744] req@ffff9d8190551880 x1862023066575872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767744.693491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066575872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767744.693496:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767744.693498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767744.693500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066575872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767744.693504:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767744.693515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767744.693521:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767744/real 1775767744] req@ffff9d8190550e00 x1862023066576000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767744.693529:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066576000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767744.693535:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767744.693537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767744.693539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:3.0:1775767744.693544:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:3.0:1775767744.693554:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767744.693559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767744/real 1775767744] req@ffff9d8190553480 x1862023066576128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767744.693566:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066576128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767744.693572:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767744.693574:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767744.693575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767745.333029:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767745.716957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767746.357047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767746.740973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767747.381009:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767747.764951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767748.405056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767748.788977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767749.429018:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767749.812976:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767749.812984:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767749.812988:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767749.812997:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4581 00000100:00100000:2.0:1775767749.813006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767749.813006:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767749.813015:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767749.813048:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066576256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767749.813080:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767749.813083:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767749.813084:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767749.813087:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4581 00000100:00080000:3.0:1775767749.813089:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767749.813091:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767749.813098:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767749.813107:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066576384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767749.813114:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767749.813117:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767749.813118:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767749.813122:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4581 00000100:00080000:3.0:1775767749.813123:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767749.813126:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767749.813129:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767749.813137:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066576512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767749.813138:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066576256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767749.813151:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767749.813153:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767749.813154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767749.813157:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4581 00000100:00080000:3.0:1775767749.813159:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767749.813162:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767749.813165:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767749.813172:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066576640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767749.813179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767749.813181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767749.813182:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767749.813184:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4581 00000100:00080000:3.0:1775767749.813194:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767749.813197:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767749.813200:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767749.813205:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767749.813209:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066576768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767749.813258:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767749/real 1775767749] req@ffff9d8188741f80 x1862023066576256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767749.813275:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066576256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767749.813283:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767749.813285:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767749.813289:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066576256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767749.813294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767749.813311:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767749.813319:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767749/real 1775767749] req@ffff9d8188743800 x1862023066576384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767749.813327:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066576384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767749.813332:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767749.813334:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767749.813336:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767749.813353:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767749.813364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767749.813369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767749/real 1775767749] req@ffff9d8188741880 x1862023066576512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767749.813382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066576512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767749.813387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767749.813388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767749.813390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767749.813394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767749.813403:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767749.813409:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767749/real 1775767749] req@ffff9d8188742680 x1862023066576640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767749.813416:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066576640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767749.813422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767749.813424:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767749.813426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767749.813429:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767749.813438:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767749.813443:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767749/real 1775767749] req@ffff9d8188742300 x1862023066576768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767749.813450:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066576768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767749.813460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767749.813462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767749.813463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066576768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767750.453088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767750.836997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767751.477047:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767751.860991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767752.501043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767752.884980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767753.525040:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767753.909019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767754.549044:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767754.932976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767754.932983:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767754.932989:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767754.932993:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767754.933001:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4586 00000100:00080000:1.0:1775767754.933010:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767754.933016:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767754.933048:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066576896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767754.933070:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767754.933073:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767754.933074:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767754.933077:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4586 00000100:00080000:1.0:1775767754.933079:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767754.933081:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767754.933088:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767754.933097:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066577024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767754.933109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767754.933112:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767754.933113:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767754.933118:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4586 00000100:00100000:0.0:1775767754.933119:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066576896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767754.933120:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767754.933126:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767754.933130:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767754.933138:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066577152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767754.933145:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767754.933148:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767754.933149:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767754.933152:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4586 00000100:00080000:1.0:1775767754.933167:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767754.933169:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767754.933173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767754.933177:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767754.933185:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066577280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767754.933193:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767754.933195:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767754.933196:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767754.933199:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4586 00000100:00080000:1.0:1775767754.933202:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767754.933204:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767754.933208:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767754.933218:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767754/real 1775767754] req@ffff9d8190553480 x1862023066576896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767754.933218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066577408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767754.933251:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066576896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767754.933262:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767754.933267:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767754.933272:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066576896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767754.933280:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767754.933301:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767754.933311:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767754/real 1775767754] req@ffff9d8190550e00 x1862023066577024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767754.933319:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066577024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767754.933325:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767754.933327:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767754.933330:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767754.933347:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767754.933359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767754.933365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767754/real 1775767754] req@ffff9d8190551880 x1862023066577152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767754.933373:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066577152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767754.933378:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767754.933380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767754.933381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767754.933385:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767754.933395:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767754.933400:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767754/real 1775767754] req@ffff9d8190551500 x1862023066577280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767754.933408:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066577280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767754.933422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767754.933425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767754.933426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767754.933430:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767754.933449:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767754.933459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767754/real 1775767754] req@ffff9d8190553100 x1862023066577408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767754.933472:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066577408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767754.933480:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767754.933482:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767754.933485:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767755.573063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767755.957010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767756.597092:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767756.980964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767757.621062:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767758.004988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767758.645063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767759.028991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767759.669043:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767760.052988:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767760.052993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767760.052996:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767760.053000:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767760.053009:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4591 00000100:00080000:1.0:1775767760.053018:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767760.053024:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767760.053058:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066577536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767760.053082:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767760.053085:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767760.053087:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767760.053089:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4591 00000100:00080000:1.0:1775767760.053091:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767760.053094:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767760.053100:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767760.053110:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066577664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767760.053117:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767760.053121:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767760.053123:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767760.053128:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066577536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767760.053128:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4591 00000100:00080000:1.0:1775767760.053134:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767760.053136:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767760.053140:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767760.053146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066577792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767760.053153:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767760.053156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767760.053157:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767760.053160:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4591 00000100:00080000:1.0:1775767760.053162:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767760.053164:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767760.053168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767760.053175:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066577920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767760.053180:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767760.053182:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767760.053184:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767760.053186:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4591 00000100:00080000:1.0:1775767760.053188:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767760.053194:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767760.053194:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767760.053200:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767760.053209:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066578048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767760.053237:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767760/real 1775767760] req@ffff9d8190553100 x1862023066577536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767760.053250:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066577536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767760.053258:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767760.053261:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767760.053263:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066577536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767760.053268:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767760.053283:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767760.053290:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767760/real 1775767760] req@ffff9d8190551500 x1862023066577664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767760.053303:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066577664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767760.053308:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767760.053310:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767760.053312:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767760.053329:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767760.053340:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767760.053346:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767760/real 1775767760] req@ffff9d8190551880 x1862023066577792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767760.053353:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066577792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767760.053358:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767760.053360:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767760.053369:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767760.053373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767760.053389:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767760.053399:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767760/real 1775767760] req@ffff9d8190550e00 x1862023066577920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767760.053412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066577920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767760.053428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767760.053432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767760.053435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066577920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767760.053442:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767760.053458:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767760.053466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767760/real 1775767760] req@ffff9d8190553480 x1862023066578048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767760.053473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066578048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767760.053480:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767760.053482:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767760.053486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767760.693049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767761.076997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767761.717019:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767762.100977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767762.741025:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767763.124983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767763.765084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767764.148996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767764.789051:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767765.172976:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767765.172979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767765.172985:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767765.172989:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767765.172998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4596 00000100:00080000:1.0:1775767765.173008:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767765.173015:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767765.173063:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066578176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767765.173096:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767765.173100:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767765.173102:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767765.173107:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4596 00000100:00080000:1.0:1775767765.173111:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767765.173115:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767765.173125:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767765.173140:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066578304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767765.173144:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066578176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767765.173156:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767765.173159:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767765.173160:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767765.173164:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4596 00000100:00080000:1.0:1775767765.173169:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767765.173172:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767765.173176:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767765.173185:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066578432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767765.173192:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767765.173195:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767765.173197:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767765.173199:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4596 00000100:00080000:1.0:1775767765.173201:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767765.173203:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767765.173207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767765.173212:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767765.173218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066578560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767765.173225:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767765.173229:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767765.173230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767765.173232:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4596 00000100:00080000:1.0:1775767765.173234:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767765.173236:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767765.173239:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767765.173247:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066578688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767765.173263:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767765/real 1775767765] req@ffff9d8190553480 x1862023066578176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767765.173283:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066578176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767765.173295:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767765.173300:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767765.173304:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066578176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767765.173326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767765.173349:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767765.173361:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767765/real 1775767765] req@ffff9d8190550e00 x1862023066578304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767765.173374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066578304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767765.173382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767765.173386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767765.173389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767765.173396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767765.173416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767765.173428:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767765/real 1775767765] req@ffff9d8190551880 x1862023066578432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767765.173441:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066578432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767765.173481:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767765.173485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767765.173489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767765.173495:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767765.173517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767765.173528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767765/real 1775767765] req@ffff9d8190551500 x1862023066578560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767765.173541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066578560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767765.173552:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767765.173555:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767765.173558:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767765.173565:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767765.173583:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767765.173594:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767765/real 1775767765] req@ffff9d8190553100 x1862023066578688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767765.173606:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066578688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767765.173615:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767765.173619:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767765.173622:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767765.813083:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767766.196984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767766.837041:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767767.220956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767767.861014:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767768.244979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767768.885024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767769.268946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767769.909081:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767770.292981:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767770.292988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767770.292992:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767770.292998:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767770.293008:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4601 00000100:00080000:1.0:1775767770.293018:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767770.293026:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767770.293063:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066578816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767770.293092:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767770.293096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767770.293098:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767770.293102:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4601 00000100:00080000:1.0:1775767770.293106:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767770.293110:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767770.293120:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767770.293135:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767770.293138:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066578816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767770.293152:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767770.293156:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767770.293158:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767770.293163:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4601 00000100:00080000:1.0:1775767770.293167:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767770.293170:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767770.293176:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767770.293187:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066579072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767770.293193:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767770.293203:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767770.293206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767770.293208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767770.293213:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4601 00000100:00080000:1.0:1775767770.293217:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767770.293220:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767770.293226:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767770.293235:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767770/real 1775767770] req@ffff9d8190553100 x1862023066578816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767770.293239:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066579200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767770.293252:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767770.293261:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767770.293263:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767770.293268:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4601 00000100:00080000:1.0:1775767770.293271:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767770.293274:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066578816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767770.293278:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767770.293288:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767770.293289:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775767770.293301:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066579328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767770.293302:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767770.293308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066578816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767770.293333:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767770.293350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767770.293359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767770/real 1775767770] req@ffff9d8190551500 x1862023066578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767770.293371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066578944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767770.293379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767770.293391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767770.293406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066578944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767770.293413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767770.293433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767770.293444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767770/real 1775767770] req@ffff9d8190551880 x1862023066579072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767770.293453:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066579072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767770.293462:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767770.293465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767770.293467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767770.293471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767770.293483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767770.293489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767770/real 1775767770] req@ffff9d8190550e00 x1862023066579200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767770.293496:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066579200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767770.293502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767770.293505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767770.293507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767770.293511:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767770.293520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767770.293526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767770/real 1775767770] req@ffff9d8190553480 x1862023066579328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767770.293533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066579328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767770.293538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767770.293540:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767770.293542:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767770.933081:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767771.316943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767771.957044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767772.340969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767772.981046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767773.364967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767774.005045:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767774.388957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767775.029112:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767775.412948:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767775.412950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767775.412957:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767775.412962:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767775.412971:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4606 00000100:00080000:1.0:1775767775.412979:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767775.412985:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767775.413039:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066579456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767775.413090:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767775.413093:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767775.413094:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767775.413097:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4606 00000100:00080000:1.0:1775767775.413100:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767775.413102:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767775.413110:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767775.413119:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066579584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767775.413126:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767775.413128:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767775.413130:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767775.413132:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4606 00000100:00080000:1.0:1775767775.413134:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767775.413136:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767775.413140:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767775.413146:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066579712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767775.413159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767775.413163:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1775767775.413164:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066579456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767775.413164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767775.413169:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4606 00000100:00080000:1.0:1775767775.413174:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767775.413177:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767775.413180:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767775.413187:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066579840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767775.413193:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767775.413195:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767775.413196:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767775.413199:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4606 00000100:00080000:1.0:1775767775.413201:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767775.413203:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767775.413206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767775.413213:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066579968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767775.413241:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767775.413280:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767775/real 1775767775] req@ffff9d8190553480 x1862023066579456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767775.413294:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066579456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767775.413301:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767775.413304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767775.413307:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066579456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767775.413313:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767775.413328:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767775.413335:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767775/real 1775767775] req@ffff9d8190550e00 x1862023066579584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767775.413343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066579584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767775.413364:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767775.413367:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767775.413369:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767775.413387:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767775.413403:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767775.413411:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767775/real 1775767775] req@ffff9d8190551880 x1862023066579712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767775.413419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066579712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767775.413428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767775.413432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767775.413435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767775.413441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767775.413461:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767775.413472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767775/real 1775767775] req@ffff9d8190551500 x1862023066579840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767775.413485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066579840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767775.413494:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767775.413498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767775.413501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767775.413507:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767775.413536:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767775.413544:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767775/real 1775767775] req@ffff9d8190553100 x1862023066579968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767775.413557:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066579968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767775.413563:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767775.413565:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767775.413567:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066579968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767776.053047:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767776.436958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767777.077060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767777.460980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767778.101093:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767778.484982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767779.125042:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767779.509009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767780.149083:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767780.532983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767780.533014:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767780.533020:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767780.533024:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767780.533031:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4612 00000100:00080000:2.0:1775767780.533041:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767780.533102:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767780.533143:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066580096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767780.533178:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767780.533181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767780.533184:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767780.533188:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4612 00000100:00080000:2.0:1775767780.533191:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767780.533197:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767780.533206:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767780.533218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767780.533225:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066580096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767780.533229:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767780.533232:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767780.533233:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767780.533237:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4612 00000100:00080000:2.0:1775767780.533240:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767780.533243:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767780.533248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767780.533257:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066580352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767780.533266:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767780.533268:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767780.533270:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767780.533274:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4612 00000100:00080000:2.0:1775767780.533276:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767780.533280:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775767780.533280:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767780.533284:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767780.533291:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066580480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767780.533315:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767780.533319:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767780.533320:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767780.533323:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4612 00000100:00080000:2.0:1775767780.533327:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767780.533330:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767780.533335:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767780.533340:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767780/real 1775767780] req@ffff9d81984a3b80 x1862023066580096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767780.533343:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066580608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767780.533360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066580096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767780.533370:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767780.533375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767780.533379:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066580096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767780.533400:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767780.533418:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767780.533427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767780/real 1775767780] req@ffff9d81984a3480 x1862023066580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767780.533435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066580224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767780.533441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767780.533443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767780.533445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767780.533449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767780.533460:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767780.533466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767780/real 1775767780] req@ffff9d81984a2d80 x1862023066580352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767780.533472:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066580352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767780.533478:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767780.533479:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767780.533481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767780.533485:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767780.533494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767780.533500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767780/real 1775767780] req@ffff9d81984a0700 x1862023066580480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767780.533506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066580480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767780.533512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767780.533514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767780.533516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767780.533520:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767780.533529:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767780.533534:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767780/real 1775767780] req@ffff9d81984a2a00 x1862023066580608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767780.533541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066580608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767780.533546:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767780.533548:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767780.533549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767781.173058:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767781.556977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767782.197048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767782.580986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767783.221089:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767783.604985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767784.245054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767784.628959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767785.269067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767785.652977:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767785.652982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767785.652984:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767785.652991:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767785.652999:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4617 00000100:00080000:2.0:1775767785.653008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767785.653014:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767785.653050:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066580736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767785.653073:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767785.653076:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767785.653078:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767785.653080:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4617 00000100:00080000:2.0:1775767785.653082:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767785.653084:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767785.653092:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767785.653101:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066580864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767785.653108:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767785.653110:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767785.653111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767785.653115:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4617 00000100:00080000:2.0:1775767785.653116:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767785.653118:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767785.653122:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767785.653128:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066580736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767785.653130:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066580992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767785.653154:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767785.653157:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767785.653159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767785.653163:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4617 00000100:00080000:2.0:1775767785.653167:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767785.653170:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767785.653176:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767785.653188:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066581120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767785.653198:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767785.653201:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767785.653203:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767785.653205:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4617 00000100:00080000:2.0:1775767785.653208:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767785.653211:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767785.653214:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767785.653221:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066581248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767785.653238:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767785.653300:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767785/real 1775767785] req@ffff9d81984a2300 x1862023066580736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767785.653320:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066580736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767785.653328:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767785.653331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767785.653334:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066580736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767785.653340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767785.653358:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767785.653368:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767785/real 1775767785] req@ffff9d81984a2a00 x1862023066580864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767785.653381:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066580864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767785.653389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767785.653393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767785.653396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767785.653436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767785.653453:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767785.653461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767785/real 1775767785] req@ffff9d81984a0700 x1862023066580992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767785.653469:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066580992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767785.653474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767785.653477:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767785.653479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066580992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767785.653482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767785.653492:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767785.653498:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767785/real 1775767785] req@ffff9d81984a2d80 x1862023066581120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767785.653510:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066581120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767785.653516:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767785.653518:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767785.653520:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767785.653523:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767785.653532:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767785.653538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767785/real 1775767785] req@ffff9d81984a3480 x1862023066581248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767785.653545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066581248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767785.653550:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767785.653551:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767785.653553:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767786.293026:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767786.676980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767787.317091:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767787.700998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767788.341061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767788.724983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767789.365000:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767789.748998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767790.389101:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767790.773004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767790.773035:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767790.773044:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767790.773048:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767790.773056:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4622 00000100:00080000:1.0:1775767790.773066:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767790.773071:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767790.773107:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066581376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767790.773133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767790.773135:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767790.773137:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767790.773139:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4622 00000100:00080000:1.0:1775767790.773142:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767790.773144:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767790.773152:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767790.773159:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066581504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767790.773166:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767790.773168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767790.773170:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767790.773173:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4622 00000100:00080000:1.0:1775767790.773176:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767790.773178:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767790.773180:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767790.773186:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066581632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767790.773192:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767790.773195:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767790.773217:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767790.773221:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4622 00000100:00080000:1.0:1775767790.773224:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767790.773226:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767790.773231:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767790.773233:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066581376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767790.773241:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066581760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767790.773248:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767790.773250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767790.773252:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767790.773255:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4622 00000100:00080000:1.0:1775767790.773256:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767790.773259:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767790.773261:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767790.773269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066581888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767790.773339:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767790.773378:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767790.773400:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767790.773425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767791.413048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767791.796964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767792.437036:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767792.820992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767793.461045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767793.845004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775767793.845202:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775767793.845245:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775767793.845263:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767793.845312:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767793.845322:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767793.845331:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775767793.845339:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767793.845383:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767790/real 1775767793] req@ffff9d8190551880 x1862023066581632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767793.845411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066581632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767793.845423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767793.845428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767793.845433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767793.845450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767790/real 1775767793] req@ffff9d8190550e00 x1862023066581760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767793.845463:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066581760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767793.845474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767793.845477:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767793.845481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767793.845488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767790/real 1775767793] req@ffff9d8190553480 x1862023066581888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767793.845499:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066581888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767793.845515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767793.845518:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767793.845522:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581888:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767793.845529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767790/real 1775767793] req@ffff9d8190553100 x1862023066581376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767793.845541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066581376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767793.845548:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767793.845552:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767793.845563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066581376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767793.845569:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767790/real 1775767793] req@ffff9d8190551500 x1862023066581504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767793.845581:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066581504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767793.845588:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767793.845592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767793.845595:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066581504:192.168.202.105@tcp:38:kworker.0 00010000:00010000:0.0:1775767794.485096:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767794.868959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767795.509057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767795.892971:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767795.892982:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767795.892987:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767795.892998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4627 00000100:00100000:3.0:1775767795.893008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767795.893012:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767795.893025:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767795.893070:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066582016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767795.893106:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767795.893110:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767795.893111:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767795.893114:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4627 00000100:00080000:1.0:1775767795.893117:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767795.893120:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767795.893130:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767795.893165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066582144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767795.893179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767795.893186:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066582016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767795.893186:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767795.893189:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767795.893199:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4627 00000100:00080000:1.0:1775767795.893203:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767795.893208:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767795.893215:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767795.893230:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066582272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767795.893242:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767795.893246:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767795.893248:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767795.893252:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4627 00000100:00080000:1.0:1775767795.893265:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767795.893269:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767795.893274:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767795.893279:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767795.893292:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066582400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767795.893305:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767795.893309:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767795.893310:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767795.893315:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4627 00000100:00080000:1.0:1775767795.893319:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767795.893324:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767795.893329:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767795.893347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767795/real 1775767795] req@ffff9d8190551500 x1862023066582016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767795.893348:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066582528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767795.893393:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066582016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767795.893407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767795.893414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767795.893419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066582016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767795.893450:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767795.893486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767795.893502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767795/real 1775767795] req@ffff9d8190553100 x1862023066582144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767795.893516:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066582144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767795.893525:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767795.893528:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767795.893531:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767795.893538:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767795.893560:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767795.893571:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767795/real 1775767795] req@ffff9d8190553480 x1862023066582272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767795.893583:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066582272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767795.893597:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767795.893600:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767795.893603:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767795.893609:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767795.893630:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767795.893641:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767795/real 1775767795] req@ffff9d8190550e00 x1862023066582400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767795.893654:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066582400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767795.893664:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767795.893668:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767795.893671:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767795.893677:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767795.893716:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767795.893729:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767795/real 1775767795] req@ffff9d8190551880 x1862023066582528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767795.893741:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066582528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767795.893750:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767795.893753:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767795.893757:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767796.533049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767796.916990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767797.557122:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767797.940947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767798.581041:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767798.964949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767799.605101:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767799.988973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767800.629015:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767801.012961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767801.012964:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767801.012971:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767801.012975:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767801.012983:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4632 00000100:00080000:1.0:1775767801.012991:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767801.012998:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767801.013034:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066582656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767801.013056:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767801.013061:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767801.013062:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767801.013065:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4632 00000100:00080000:1.0:1775767801.013067:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767801.013070:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767801.013077:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767801.013088:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066582784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767801.013126:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767801.013131:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767801.013133:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767801.013139:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4632 00000100:00080000:1.0:1775767801.013143:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767801.013149:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767801.013150:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066582656:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775767801.013158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767801.013173:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066582912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767801.013181:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767801.013183:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767801.013185:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767801.013188:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4632 00000100:00080000:1.0:1775767801.013199:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767801.013202:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767801.013207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767801.013208:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767801.013219:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066583040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767801.013226:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767801.013229:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767801.013230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767801.013232:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4632 00000100:00080000:1.0:1775767801.013234:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767801.013237:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767801.013241:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767801.013248:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066583168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767801.013260:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767801/real 1775767801] req@ffff9d8190551880 x1862023066582656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767801.013276:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066582656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767801.013283:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767801.013287:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767801.013290:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066582656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767801.013295:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767801.013308:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767801.013316:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767801/real 1775767801] req@ffff9d8190550e00 x1862023066582784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767801.013324:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066582784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767801.013329:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767801.013331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767801.013333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767801.013351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767801.013362:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767801.013367:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767801/real 1775767801] req@ffff9d8190553480 x1862023066582912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767801.013374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066582912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767801.013379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767801.013381:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767801.013383:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066582912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767801.013386:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767801.013396:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767801.013401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767801/real 1775767801] req@ffff9d8190553100 x1862023066583040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767801.013408:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066583040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767801.013414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767801.013416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767801.013418:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767801.013421:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767801.013430:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767801.013435:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767801/real 1775767801] req@ffff9d8190551500 x1862023066583168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767801.013442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066583168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767801.013447:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767801.013449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767801.013450:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767801.653071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767802.037009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767802.677140:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767803.061011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767803.701084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767804.084987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767804.725056:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767805.108981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767805.749087:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767806.132971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767806.133068:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767806.133076:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767806.133081:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767806.133092:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4637 00000100:00080000:2.0:1775767806.133104:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767806.133110:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767806.133157:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066583296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767806.133186:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767806.133189:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767806.133190:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767806.133193:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4637 00000100:00080000:2.0:1775767806.133195:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767806.133198:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767806.133207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767806.133218:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066583424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767806.133225:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767806.133229:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767806.133230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767806.133234:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4637 00000100:00080000:2.0:1775767806.133238:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767806.133243:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066583296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767806.133244:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767806.133255:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767806.133264:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066583552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767806.133275:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767806.133279:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767806.133281:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767806.133286:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4637 00000100:00080000:2.0:1775767806.133308:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767806.133316:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767806.133317:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767806.133326:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767806.133349:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066583680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767806.133357:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767806.133360:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767806.133363:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767806.133364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767806/real 1775767806] req@ffff9d81984a3b80 x1862023066583296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767806.133385:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4637 00000100:00100000:0.0:1775767806.133390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066583296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767806.133393:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767806.133407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775767806.133409:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775767806.133416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767806.133421:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066583296:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775767806.133421:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767806.133434:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775767806.133441:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066583808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767806.133459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767806.133475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767806/real 1775767806] req@ffff9d81984a0000 x1862023066583424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767806.133488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066583424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767806.133494:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767806.133496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767806.133498:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767806.133517:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767806.133528:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767806.133533:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767806/real 1775767806] req@ffff9d81984a2680 x1862023066583552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767806.133540:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066583552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767806.133545:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767806.133547:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767806.133549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767806.133553:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767806.133563:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767806.133568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767806/real 1775767806] req@ffff9d81984a0e00 x1862023066583680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767806.133575:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066583680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767806.133586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767806.133588:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767806.133590:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767806.133593:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767806.133603:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767806.133608:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767806/real 1775767806] req@ffff9d81984a1f80 x1862023066583808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767806.133615:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066583808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767806.133620:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767806.133622:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767806.133624:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066583808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767806.773259:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767807.157018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767807.797093:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767808.180988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767808.821083:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767809.204987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767809.845040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767810.228987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767810.869040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767811.252970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767811.252982:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767811.252988:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767811.252992:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767811.253015:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4642 00000100:00080000:1.0:1775767811.253024:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767811.253030:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767811.253062:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066583936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767811.253084:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767811.253087:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767811.253088:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767811.253091:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4642 00000100:00080000:1.0:1775767811.253094:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767811.253097:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767811.253104:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767811.253113:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066584064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767811.253121:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767811.253123:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767811.253125:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767811.253128:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4642 00000100:00080000:1.0:1775767811.253131:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767811.253133:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066583936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767811.253135:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767811.253142:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767811.253151:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066584192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767811.253158:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767811.253161:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767811.253163:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767811.253165:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4642 00000100:00080000:1.0:1775767811.253180:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767811.253183:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767811.253187:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767811.253191:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767811.253199:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066584320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767811.253211:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767811.253213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767811.253215:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767811.253218:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4642 00000100:00080000:1.0:1775767811.253221:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767811.253223:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767811.253226:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767811.253231:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767811/real 1775767811] req@ffff9d8190551500 x1862023066583936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767811.253236:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066584448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767811.253262:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066583936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767811.253275:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767811.253278:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767811.253280:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066583936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767811.253286:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767811.253300:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767811.253307:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767811/real 1775767811] req@ffff9d8190553100 x1862023066584064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767811.253316:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066584064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767811.253321:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767811.253323:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767811.253325:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767811.253352:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767811.253363:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767811.253370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767811/real 1775767811] req@ffff9d8190553480 x1862023066584192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767811.253377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066584192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767811.253383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767811.253385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767811.253387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767811.253391:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767811.253401:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767811.253406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767811/real 1775767811] req@ffff9d8190550e00 x1862023066584320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767811.253413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066584320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767811.253420:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767811.253421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767811.253423:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767811.253426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767811.253436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767811.253441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767811/real 1775767811] req@ffff9d8190551880 x1862023066584448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767811.253448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066584448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767811.253453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767811.253455:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767811.253456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767811.893090:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767812.277056:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767812.917101:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767813.301026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767813.941069:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767814.325007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767814.965021:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767815.348944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767815.989043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767816.372986:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767816.372998:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1775767816.372998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767816.373002:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767816.373012:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4647 00000100:00080000:1.0:1775767816.373020:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767816.373027:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767816.373061:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066584576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767816.373085:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767816.373087:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767816.373089:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767816.373091:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4647 00000100:00080000:1.0:1775767816.373094:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767816.373096:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767816.373103:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767816.373111:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066584704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767816.373118:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767816.373121:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767816.373122:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767816.373125:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4647 00000100:00080000:1.0:1775767816.373127:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767816.373130:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767816.373132:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767816.373136:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066584576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767816.373142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066584832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767816.373149:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767816.373152:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767816.373153:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767816.373156:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4647 00000100:00080000:1.0:1775767816.373158:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767816.373160:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767816.373163:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767816.373171:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066584960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767816.373177:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767816.373179:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767816.373181:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767816.373184:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4647 00000100:00080000:1.0:1775767816.373186:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767816.373190:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767816.373191:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767816.373197:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767816.373206:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066585088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767816.373243:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767816/real 1775767816] req@ffff9d8190551880 x1862023066584576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767816.373263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066584576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767816.373274:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767816.373279:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767816.373284:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066584576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767816.373292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767816.373317:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767816.373330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767816/real 1775767816] req@ffff9d8190550e00 x1862023066584704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767816.373343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066584704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767816.373351:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767816.373354:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767816.373358:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767816.373381:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767816.373400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767816.373411:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767816/real 1775767816] req@ffff9d8190553480 x1862023066584832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767816.373423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066584832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767816.373429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767816.373432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767816.373436:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767816.373442:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767816.373464:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767816.373475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767816/real 1775767816] req@ffff9d8190553100 x1862023066584960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767816.373487:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066584960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767816.373499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767816.373502:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767816.373505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066584960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767816.373511:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767816.373528:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767816.373537:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767816/real 1775767816] req@ffff9d8190551500 x1862023066585088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767816.373555:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066585088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767816.373570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767816.373574:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767816.373577:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767817.013126:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767817.397000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767818.037099:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767818.421000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767819.061076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767819.445004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767820.085057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767820.469023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767821.109024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767821.493008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767821.493013:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767821.493020:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767821.493025:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767821.493035:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4653 00000100:00080000:1.0:1775767821.493046:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767821.493060:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767821.493100:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066585216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767821.493131:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767821.493134:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767821.493136:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767821.493139:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4653 00000100:00080000:1.0:1775767821.493142:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767821.493145:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767821.493152:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767821.493164:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066585344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767821.493178:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767821.493181:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066585216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767821.493182:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767821.493185:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767821.493192:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4653 00000100:00080000:1.0:1775767821.493194:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767821.493196:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767821.493199:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767821.493209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066585472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767821.493215:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767821.493218:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767821.493219:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767821.493222:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4653 00000100:00080000:1.0:1775767821.493231:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767821.493234:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767821.493237:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767821.493241:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767821.493246:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066585600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767821.493253:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767821.493256:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767821.493257:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767821.493260:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4653 00000100:00080000:1.0:1775767821.493263:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767821.493265:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767821.493269:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767821.493280:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066585728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767821.493286:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767821/real 1775767821] req@ffff9d8190551500 x1862023066585216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767821.493309:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066585216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767821.493316:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767821.493320:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767821.493324:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066585216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767821.493342:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767821.493363:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767821.493371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767821/real 1775767821] req@ffff9d8190553100 x1862023066585344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767821.493379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066585344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767821.493389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767821.493391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767821.493393:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767821.493397:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767821.493407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767821.493412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767821/real 1775767821] req@ffff9d8190553480 x1862023066585472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767821.493421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066585472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767821.493430:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767821.493433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767821.493437:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767821.493443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767821.493463:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767821.493475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767821/real 1775767821] req@ffff9d8190550e00 x1862023066585600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767821.493488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066585600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767821.493494:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767821.493496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767821.493499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767821.493503:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767821.493517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767821.493524:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767821/real 1775767821] req@ffff9d8190551880 x1862023066585728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767821.493531:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066585728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767821.493537:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767821.493538:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767821.493540:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767822.133031:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767822.517031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767823.157084:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767823.541084:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767824.181043:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767824.564993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767825.205076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767825.588971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767826.229065:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767826.613014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767826.613025:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767826.613031:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767826.613035:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767826.613043:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4658 00000100:00080000:2.0:1775767826.613052:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767826.613058:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767826.613091:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066585856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767826.613117:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767826.613119:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767826.613121:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767826.613123:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4658 00000100:00080000:2.0:1775767826.613125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767826.613128:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767826.613134:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767826.613142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066585984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767826.613149:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767826.613151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767826.613152:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767826.613154:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4658 00000100:00080000:2.0:1775767826.613156:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767826.613158:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767826.613161:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767826.613167:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066586112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767826.613190:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767826.613194:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066585856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767826.613195:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767826.613201:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767826.613207:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4658 00000100:00080000:2.0:1775767826.613211:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767826.613215:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767826.613221:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767826.613235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066586240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767826.613242:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767826.613245:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767826.613246:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767826.613248:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4658 00000100:00080000:2.0:1775767826.613252:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767826.613256:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775767826.613258:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767826.613264:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767826.613273:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066586368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767826.613310:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767826/real 1775767826] req@ffff9d81984a1180 x1862023066585856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767826.613325:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066585856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767826.613332:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767826.613335:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767826.613338:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066585856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767826.613344:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767826.613359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767826.613366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767826/real 1775767826] req@ffff9d81984a1880 x1862023066585984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767826.613374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066585984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767826.613380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767826.613382:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767826.613384:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066585984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767826.613401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767826.613410:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767826.613423:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767826/real 1775767826] req@ffff9d81984a3800 x1862023066586112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767826.613430:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066586112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767826.613435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767826.613437:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767826.613439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767826.613443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767826.613454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767826.613460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767826/real 1775767826] req@ffff9d81984a3100 x1862023066586240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767826.613467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066586240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767826.613474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767826.613475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767826.613477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767826.613480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767826.613490:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767826.613496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767826/real 1775767826] req@ffff9d81984a1500 x1862023066586368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767826.613503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066586368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767826.613508:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767826.613509:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767826.613511:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767827.253015:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767827.636944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767828.277073:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767828.660981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767829.301083:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767829.685007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767830.325051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767830.708962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767831.349082:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767831.733003:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1775767831.733006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767831.733018:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767831.733025:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767831.733037:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4663 00000100:00080000:2.0:1775767831.733051:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767831.733059:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767831.733105:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066586496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767831.733141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767831.733146:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767831.733148:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767831.733153:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4663 00000100:00080000:2.0:1775767831.733157:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767831.733162:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767831.733174:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767831.733191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066586624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767831.733203:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767831.733209:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767831.733210:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066586496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767831.733211:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767831.733220:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4663 00000100:00080000:2.0:1775767831.733224:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767831.733230:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767831.733236:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767831.733253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066586752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767831.733269:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767831.733273:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767831.733275:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767831.733280:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4663 00000100:00080000:2.0:1775767831.733284:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767831.733288:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767831.733294:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767831.733297:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767831.733310:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066586880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767831.733321:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767831.733325:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767831.733327:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767831.733331:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4663 00000100:00080000:2.0:1775767831.733335:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767831.733339:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767831.733343:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767831.733354:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066587008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767831.733365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767831/real 1775767831] req@ffff9d81984a0a80 x1862023066586496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767831.733389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066586496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767831.733397:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767831.733401:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767831.733404:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066586496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767831.733426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767831.733445:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767831.733455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767831/real 1775767831] req@ffff9d81984a1500 x1862023066586624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767831.733463:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066586624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767831.733469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767831.733472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767831.733474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767831.733478:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767831.733492:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767831.733502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767831/real 1775767831] req@ffff9d81984a3100 x1862023066586752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767831.733515:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066586752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767831.733531:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767831.733535:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767831.733539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767831.733545:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767831.733563:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767831.733570:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767831/real 1775767831] req@ffff9d81984a3800 x1862023066586880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767831.733578:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066586880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767831.733585:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767831.733587:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767831.733590:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066586880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767831.733596:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767831.733615:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767831.733626:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767831/real 1775767831] req@ffff9d81984a1880 x1862023066587008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767831.733639:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066587008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767831.733649:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767831.733652:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767831.733655:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767832.373092:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767832.756972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767833.397035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767833.780960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767834.421086:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767834.804980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767835.445006:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767835.828978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767836.469050:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767836.852951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767836.852958:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767836.852964:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767836.852968:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767836.852976:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4668 00000100:00080000:1.0:1775767836.852985:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767836.852991:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767836.853025:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066587136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767836.853049:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767836.853053:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767836.853055:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767836.853059:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4668 00000100:00080000:1.0:1775767836.853063:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767836.853068:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767836.853079:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767836.853094:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066587264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767836.853100:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066587136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767836.853106:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767836.853112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767836.853113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767836.853117:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4668 00000100:00080000:1.0:1775767836.853120:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767836.853124:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767836.853128:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767836.853137:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066587392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767836.853144:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767836.853146:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767836.853147:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767836.853150:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4668 00000100:00080000:1.0:1775767836.853153:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767836.853155:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767836.853158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767836.853163:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767836.853168:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066587520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767836.853175:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767836.853178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767836.853179:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767836.853182:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4668 00000100:00080000:1.0:1775767836.853184:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767836.853186:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767836.853190:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767836.853198:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066587648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767836.853207:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767836/real 1775767836] req@ffff9d8190551880 x1862023066587136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767836.853230:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066587136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767836.853239:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767836.853242:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767836.853245:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066587136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767836.853262:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767836.853279:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767836.853287:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767836/real 1775767836] req@ffff9d8190550e00 x1862023066587264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767836.853296:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066587264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767836.853301:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767836.853303:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767836.853305:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767836.853311:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767836.853321:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767836.853327:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767836/real 1775767836] req@ffff9d8190553480 x1862023066587392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767836.853335:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066587392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767836.853340:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767836.853342:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767836.853344:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767836.853347:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767836.853357:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767836.853362:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767836/real 1775767836] req@ffff9d8190553100 x1862023066587520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767836.853369:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066587520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767836.853375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767836.853377:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767836.853379:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767836.853382:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767836.853400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767836.853409:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767836/real 1775767836] req@ffff9d8190551500 x1862023066587648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767836.853417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066587648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767836.853423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767836.853425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767836.853426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767837.493015:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767837.876979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767838.517084:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767838.900963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767839.541023:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767839.924967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767840.565132:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767840.948992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767841.589024:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767841.972942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767841.973019:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767841.973030:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767841.973036:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767841.973048:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4673 00000100:00080000:1.0:1775767841.973062:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767841.973072:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767841.973149:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066587776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767841.973189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767841.973193:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767841.973195:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767841.973199:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4673 00000100:00080000:1.0:1775767841.973204:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767841.973209:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767841.973223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767841.973239:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767841.973253:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066587776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767841.973255:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767841.973277:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767841.973281:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767841.973290:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4673 00000800:00000100:0.0:1775767841.973353:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767841.973394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767841/real 1775767841] req@ffff9d8190551500 x1862023066587776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767841.973410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066587776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767841.973420:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767841.973421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767841.973427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767841.973432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066587776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767841.973432:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767841.973446:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767841.973456:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775767841.973475:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066588032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767841.973479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767841.973491:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767841/real 1775767841] req@ffff9d8190553100 x1862023066587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767841.973493:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767841.973501:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767841.973504:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767841.973513:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4673 00000100:00080000:1.0:1775767841.973519:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767841.973523:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767841.973526:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066587904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767841.973529:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775767841.973538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767841.973543:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775767841.973543:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066588160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767841.973550:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066587904:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775767841.973563:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767841.973566:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775767841.973568:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588032:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775767841.973569:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767841.973577:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4673 00000100:00080000:1.0:1775767841.973581:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767841.973585:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767841.973590:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775767841.973593:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767841.973599:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767841/real 1775767841] req@ffff9d8190553480 x1862023066588032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767841.973607:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066588032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767841.973626:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066588288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767841.973627:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767841.973637:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767841.973649:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767841.973663:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767841.973686:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767841.973698:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767841/real 1775767841] req@ffff9d8190550e00 x1862023066588160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767841.973742:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066588160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767841.973758:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767841.973762:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767841.973766:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767841.973779:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767841.973796:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767841.973803:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767841/real 1775767841] req@ffff9d8190551880 x1862023066588288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767841.973811:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066588288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767841.973816:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767841.973818:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767841.973821:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767842.613038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767842.996970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767843.637059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767844.020971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767844.661046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767845.044994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767845.685104:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767846.069052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767846.709003:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767847.092948:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767847.092948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767847.092957:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767847.092961:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767847.092977:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4678 00000100:00080000:1.0:1775767847.092988:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767847.092996:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767847.093036:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066588416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767847.093062:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767847.093065:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767847.093066:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767847.093069:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4678 00000100:00080000:1.0:1775767847.093072:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767847.093082:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767847.093090:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767847.093100:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066588544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767847.093107:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767847.093110:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767847.093111:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767847.093114:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4678 00000100:00080000:1.0:1775767847.093117:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767847.093119:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767847.093123:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767847.093130:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066588672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767847.093138:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066588416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767847.093138:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767847.093143:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767847.093147:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767847.093150:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4678 00000100:00080000:1.0:1775767847.093152:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767847.093154:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767847.093158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767847.093167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066588800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767847.093173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767847.093176:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767847.093177:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767847.093180:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4678 00000100:00080000:1.0:1775767847.093193:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767847.093195:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767847.093198:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767847.093203:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767847.093210:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066588928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767847.093244:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767847/real 1775767847] req@ffff9d8190551880 x1862023066588416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767847.093259:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066588416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767847.093271:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767847.093274:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767847.093277:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066588416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767847.093282:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767847.093297:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767847.093305:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767847/real 1775767847] req@ffff9d8190550e00 x1862023066588544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767847.093312:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066588544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767847.093318:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767847.093320:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767847.093322:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767847.093340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767847.093350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767847.093356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767847/real 1775767847] req@ffff9d8190553480 x1862023066588672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767847.093363:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066588672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767847.093369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767847.093371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767847.093373:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767847.093377:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767847.093387:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767847.093393:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767847/real 1775767847] req@ffff9d8190553100 x1862023066588800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767847.093400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066588800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767847.093407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767847.093408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767847.093410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767847.093414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767847.093424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767847.093429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767847/real 1775767847] req@ffff9d8190551500 x1862023066588928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767847.093437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066588928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767847.093442:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767847.093444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767847.093446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066588928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767847.733106:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767848.117019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767848.757018:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767849.140954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767849.781113:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767850.165042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767850.805063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767851.189001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767851.829077:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767852.212964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767852.212966:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767852.212974:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767852.212978:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767852.212986:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4683 00000100:00080000:1.0:1775767852.212995:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767852.213002:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767852.213036:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066589056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767852.213060:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767852.213063:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767852.213064:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767852.213067:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4683 00000100:00080000:1.0:1775767852.213075:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767852.213078:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767852.213090:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767852.213105:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767852.213113:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066589056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767852.213121:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767852.213125:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767852.213127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767852.213133:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4683 00000100:00080000:1.0:1775767852.213138:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767852.213143:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767852.213149:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767852.213162:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066589312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767852.213172:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767852.213181:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767852.213184:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767852.213185:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767852.213190:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4683 00000100:00080000:1.0:1775767852.213195:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767852.213198:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767852.213202:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767852.213211:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066589440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767852.213218:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767852.213220:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767852.213222:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767852.213224:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4683 00000100:00080000:1.0:1775767852.213226:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767852.213227:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767852.213230:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767852.213240:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066589568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767852.213242:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767852/real 1775767852] req@ffff9d8190551500 x1862023066589056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767852.213278:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066589056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767852.213287:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767852.213290:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767852.213293:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066589056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767852.213311:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767852.213329:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767852.213337:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767852/real 1775767852] req@ffff9d8190553100 x1862023066589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767852.213346:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066589184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767852.213351:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767852.213353:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767852.213356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767852.213359:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767852.213369:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767852.213374:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767852/real 1775767852] req@ffff9d8190553480 x1862023066589312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767852.213382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066589312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767852.213387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767852.213389:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767852.213391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767852.213394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767852.213404:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767852.213409:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767852/real 1775767852] req@ffff9d8190550e00 x1862023066589440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767852.213417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066589440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767852.213423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767852.213425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767852.213427:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767852.213430:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767852.213439:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767852.213444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767852/real 1775767852] req@ffff9d8190551880 x1862023066589568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767852.213451:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066589568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767852.213461:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767852.213463:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767852.213465:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767852.853075:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767853.236987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767853.877047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767854.260947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767854.901038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767855.285019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767855.925067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767856.309013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767856.949064:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767857.332999:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775767857.333005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767857.333015:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767857.333022:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767857.333034:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4688 00000100:00080000:1.0:1775767857.333049:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767857.333057:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767857.333105:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066589696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767857.333134:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767857.333137:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767857.333139:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767857.333142:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4688 00000100:00080000:1.0:1775767857.333144:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767857.333147:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767857.333157:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767857.333165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066589824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767857.333173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767857.333176:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767857.333177:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767857.333181:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4688 00000100:00080000:1.0:1775767857.333183:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767857.333187:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767857.333190:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767857.333199:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066589952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767857.333201:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066589696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767857.333206:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767857.333212:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767857.333213:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767857.333216:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4688 00000100:00080000:1.0:1775767857.333218:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767857.333219:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767857.333222:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767857.333231:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066590080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767857.333236:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767857.333238:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767857.333239:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767857.333241:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4688 00000100:00080000:1.0:1775767857.333243:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767857.333245:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767857.333248:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767857.333257:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066590208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767857.333355:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767857.333411:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767857.333439:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767857.333467:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767857.973081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767858.357066:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767858.997050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767859.381011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767860.021068:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767860.404990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767860.405169:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767860.405211:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767860.405229:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767860.405265:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767860.405278:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767860.405290:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767860.405300:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767860.405350:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767857/real 1775767860] req@ffff9d8190553480 x1862023066589952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767860.405386:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066589952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767860.405398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767860.405403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767860.405407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767860.405414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767857/real 1775767860] req@ffff9d8190553100 x1862023066590080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767860.405425:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066590080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767860.405431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767860.405433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767860.405436:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767860.405441:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767857/real 1775767860] req@ffff9d8190551500 x1862023066590208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767860.405450:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066590208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767860.405458:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767860.405462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767860.405465:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590208:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767860.405472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767857/real 1775767860] req@ffff9d8190551880 x1862023066589696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767860.405488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066589696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767860.405497:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767860.405501:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767860.405504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066589696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767860.405514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767857/real 1775767860] req@ffff9d8190550e00 x1862023066589824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767860.405542:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066589824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767860.405550:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767860.405554:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767860.405557:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066589824:192.168.202.105@tcp:38:kworker.0 00010000:00010000:3.0:1775767861.045034:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767861.428974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767862.069072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767862.452967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767862.453031:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767862.453039:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767862.453043:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767862.453052:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4693 00000100:00080000:1.0:1775767862.453063:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767862.453069:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767862.453106:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066590336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767862.453139:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767862.453143:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767862.453145:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767862.453150:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4693 00000100:00080000:1.0:1775767862.453154:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767862.453157:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767862.453167:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767862.453197:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066590464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767862.453210:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066590336:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775767862.453210:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767862.453218:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767862.453221:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767862.453226:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4693 00000100:00080000:1.0:1775767862.453231:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767862.453235:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767862.453241:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767862.453257:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066590592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767862.453267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767862.453270:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767862.453272:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775767862.453277:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767862.453280:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4693 00000100:00080000:1.0:1775767862.453285:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767862.453290:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767862.453296:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767862.453311:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066590720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767862.453322:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767862.453325:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767862.453328:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767862.453330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767862/real 1775767862] req@ffff9d8190550e00 x1862023066590336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767862.453336:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4693 00000100:00080000:1.0:1775767862.453343:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767862.453349:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767862.453355:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767862.453359:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066590336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767862.453371:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066590848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767862.453373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767862.453383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767862.453390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066590336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767862.453418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767862.453441:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767862.453455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767862/real 1775767862] req@ffff9d8190551880 x1862023066590464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767862.453473:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066590464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767862.453481:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767862.453484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767862.453487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767862.453492:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767862.453543:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767862.453555:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767862/real 1775767862] req@ffff9d8190551500 x1862023066590592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767862.453566:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066590592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767862.453574:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767862.453578:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767862.453582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767862.453588:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767862.453609:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767862.453620:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767862/real 1775767862] req@ffff9d8190553100 x1862023066590720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767862.453634:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066590720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767862.453651:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767862.453655:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767862.453659:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767862.453666:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767862.453687:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767862.453699:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767862/real 1775767862] req@ffff9d8190553480 x1862023066590848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767862.453712:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066590848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767862.453722:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767862.453725:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767862.453729:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066590848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767863.093048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767863.476990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767864.117066:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767864.500950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767865.141065:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767865.524962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767866.165028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767866.548930:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767867.189041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767867.572990:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767867.573000:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775767867.573003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767867.573005:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767867.573016:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4699 00000100:00080000:3.0:1775767867.573029:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767867.573036:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767867.573075:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066590976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767867.573104:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767867.573108:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767867.573110:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767867.573114:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4699 00000100:00080000:3.0:1775767867.573118:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767867.573122:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767867.573131:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767867.573146:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066591104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767867.573160:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066590976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767867.573172:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767867.573176:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767867.573179:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767867.573185:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4699 00000100:00080000:3.0:1775767867.573190:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767867.573194:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767867.573200:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767867.573212:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066591232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767867.573224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767867.573224:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767867.573231:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767867.573234:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767867.573240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4699 00000100:00080000:3.0:1775767867.573245:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767867.573249:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767867.573256:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767867.573270:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066591360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767867.573276:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767867/real 1775767867] req@ffff9d8188740e00 x1862023066590976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767867.573288:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767867.573296:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767867.573299:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767867.573304:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4699 00000100:00100000:0.0:1775767867.573308:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066590976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767867.573310:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767867.573319:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767867.573327:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767867.573338:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767867.573344:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775767867.573346:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066591488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767867.573351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066590976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767867.573383:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767867.573405:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767867.573418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767867/real 1775767867] req@ffff9d8188741180 x1862023066591104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767867.573431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066591104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767867.573440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767867.573443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767867.573447:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767867.573464:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767867.573483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767867.573493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767867/real 1775767867] req@ffff9d8188743100 x1862023066591232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767867.573504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066591232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767867.573513:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767867.573516:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767867.573520:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767867.573525:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767867.573542:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767867.573550:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767867/real 1775767867] req@ffff9d8188743480 x1862023066591360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767867.573561:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066591360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767867.573570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767867.573573:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767867.573576:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767867.573581:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767867.573598:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767867.573606:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767867/real 1775767867] req@ffff9d8188743b80 x1862023066591488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767867.573618:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066591488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767867.573626:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767867.573630:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767867.573633:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767868.213026:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767868.597039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767869.237011:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767869.620965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767870.261085:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767870.644979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767871.285040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767871.668962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767872.309061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767872.692994:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767872.692997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767872.693004:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767872.693008:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767872.693016:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4704 00000100:00080000:3.0:1775767872.693024:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767872.693031:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767872.693081:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066591616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767872.693109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767872.693112:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767872.693114:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767872.693117:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4704 00000100:00080000:3.0:1775767872.693120:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767872.693124:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767872.693132:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767872.693140:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066591616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767872.693156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066591744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767872.693165:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767872.693168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767872.693170:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767872.693173:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4704 00000100:00080000:3.0:1775767872.693176:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767872.693178:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767872.693181:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767872.693189:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066591872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767872.693196:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767872.693198:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767872.693199:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767872.693202:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4704 00000800:00000100:0.0:1775767872.693206:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767872.693207:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767872.693211:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767872.693215:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767872.693224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066592000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767872.693230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767872.693233:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767872.693234:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767872.693238:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4704 00000100:00080000:3.0:1775767872.693241:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767872.693244:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767872.693248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767872.693250:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767872/real 1775767872] req@ffff9d8188741500 x1862023066591616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767872.693258:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066592128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767872.693270:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066591616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767872.693279:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767872.693282:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767872.693286:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066591616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767872.693312:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767872.693326:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767872.693334:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767872/real 1775767872] req@ffff9d8188740a80 x1862023066591744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767872.693342:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066591744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767872.693347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767872.693349:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767872.693351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767872.693355:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767872.693365:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767872.693370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767872/real 1775767872] req@ffff9d8188742d80 x1862023066591872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767872.693377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066591872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767872.693383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767872.693384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767872.693386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066591872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767872.693389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767872.693400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767872.693434:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767872/real 1775767872] req@ffff9d8188743b80 x1862023066592000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767872.693443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066592000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767872.693449:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767872.693452:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767872.693453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767872.693457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767872.693471:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767872.693478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767872/real 1775767872] req@ffff9d8188743480 x1862023066592128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767872.693485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066592128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767872.693491:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767872.693492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767872.693494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767873.333044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767873.716947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767874.357063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767874.740990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767875.381093:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767875.764942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767876.405019:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767876.788983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767877.429077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767877.812979:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767877.812987:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767877.812991:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767877.813001:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4709 00000100:00080000:3.0:1775767877.813010:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767877.813017:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767877.813036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:3.0:1775767877.813050:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066592256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767877.813066:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767877.813069:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767877.813070:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767877.813074:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4709 00000100:00080000:3.0:1775767877.813076:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767877.813081:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066592256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767877.813081:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767877.813093:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767877.813105:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066592384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767877.813113:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767877.813116:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767877.813118:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767877.813121:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4709 00000100:00080000:3.0:1775767877.813132:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767877.813135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767877.813139:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767877.813143:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767877.813153:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066592512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767877.813161:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767877.813163:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767877.813164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767877.813166:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4709 00000100:00080000:3.0:1775767877.813169:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767877.813172:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767877.813176:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767877.813186:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767877/real 1775767877] req@ffff9d8188743100 x1862023066592256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767877.813186:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066592640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767877.813221:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767877.813224:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767877.813225:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767877.813228:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4709 00000100:00100000:0.0:1775767877.813231:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066592256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767877.813232:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767877.813239:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767877.813247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775767877.813250:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775767877.813256:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066592768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767877.813257:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767877.813266:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066592256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767877.813289:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767877.813305:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767877.813313:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767877/real 1775767877] req@ffff9d8188741180 x1862023066592384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767877.813321:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066592384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767877.813326:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767877.813329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767877.813331:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767877.813335:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767877.813345:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767877.813350:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767877/real 1775767877] req@ffff9d8188740e00 x1862023066592512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767877.813376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066592512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767877.813382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767877.813384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767877.813386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767877.813391:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767877.813408:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767877.813420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767877/real 1775767877] req@ffff9d8188742300 x1862023066592640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767877.813433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066592640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767877.813443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767877.813446:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767877.813450:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767877.813456:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767877.813475:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767877.813483:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767877/real 1775767877] req@ffff9d8188742680 x1862023066592768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767877.813491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066592768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767877.813497:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767877.813499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767877.813501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066592768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767878.453031:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767878.836975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767879.477031:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767879.860981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767880.501033:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767880.884974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767881.525025:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767881.908978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767882.549034:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767882.932979:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767882.932981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767882.932989:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767882.932993:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767882.933000:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4714 00000100:00080000:3.0:1775767882.933009:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767882.933015:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767882.933048:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066592896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767882.933071:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767882.933074:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767882.933075:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767882.933078:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4714 00000100:00080000:3.0:1775767882.933080:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767882.933082:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767882.933089:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767882.933098:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066593024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767882.933106:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767882.933109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767882.933110:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767882.933115:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4714 00000100:00100000:0.0:1775767882.933116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066592896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767882.933118:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767882.933125:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767882.933128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767882.933136:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066593152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767882.933142:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767882.933146:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767882.933148:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767882.933152:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4714 00000100:00080000:3.0:1775767882.933167:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767882.933172:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767882.933178:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767882.933182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767882.933198:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066593280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767882.933208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767882.933212:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767882.933214:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767882.933219:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4714 00000100:00100000:0.0:1775767882.933224:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767882/real 1775767882] req@ffff9d8188741880 x1862023066592896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767882.933226:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767882.933234:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767882.933245:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767882.933255:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066592896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767882.933261:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066593408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767882.933273:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767882.933280:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767882.933285:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066592896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767882.933329:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767882.933349:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767882.933359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767882/real 1775767882] req@ffff9d8188743800 x1862023066593024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767882.933368:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066593024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767882.933373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767882.933376:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767882.933378:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767882.933382:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767882.933393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767882.933398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767882/real 1775767882] req@ffff9d8188741f80 x1862023066593152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767882.933410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066593152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767882.933415:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767882.933417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767882.933419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767882.933423:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767882.933433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767882.933438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767882/real 1775767882] req@ffff9d8188740000 x1862023066593280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767882.933445:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066593280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767882.933452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767882.933453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767882.933455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767882.933458:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767882.933467:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767882.933472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767882/real 1775767882] req@ffff9d8188740700 x1862023066593408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767882.933479:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066593408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767882.933489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767882.933491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767882.933493:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767883.573061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767883.956986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767884.597066:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767884.981010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767885.621020:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767886.004944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767886.645148:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767887.028978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767887.669049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767888.052956:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767888.052962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767888.052963:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767888.052968:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767888.052976:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4719 00000100:00080000:2.0:1775767888.052985:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767888.052990:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767888.053023:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066593536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767888.053045:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767888.053047:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767888.053049:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767888.053051:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4719 00000100:00080000:2.0:1775767888.053054:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767888.053056:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767888.053063:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767888.053072:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066593664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767888.053080:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767888.053084:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767888.053086:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066593536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767888.053086:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767888.053092:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4719 00000100:00080000:2.0:1775767888.053096:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767888.053099:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767888.053102:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767888.053109:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066593792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767888.053114:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767888.053117:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767888.053118:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767888.053121:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4719 00000100:00080000:2.0:1775767888.053123:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767888.053125:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767888.053134:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767888.053142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066593920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767888.053148:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767888.053150:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767888.053152:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767888.053156:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4719 00000800:00000100:0.0:1775767888.053158:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775767888.053159:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767888.053164:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767888.053167:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767888.053174:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066594048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767888.053218:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767888/real 1775767888] req@ffff9d81984a1180 x1862023066593536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767888.053239:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066593536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767888.053251:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767888.053271:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767888.053276:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066593536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767888.053301:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767888.053326:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767888.053339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767888/real 1775767888] req@ffff9d81984a1f80 x1862023066593664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767888.053352:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066593664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767888.053360:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767888.053364:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767888.053367:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767888.053373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767888.053394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767888.053405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767888/real 1775767888] req@ffff9d81984a0e00 x1862023066593792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767888.053418:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066593792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767888.053426:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767888.053430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767888.053433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767888.053439:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767888.053459:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767888.053470:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767888/real 1775767888] req@ffff9d81984a2680 x1862023066593920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767888.053482:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066593920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767888.053492:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767888.053496:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767888.053500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066593920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767888.053506:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767888.053527:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767888.053537:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767888/real 1775767888] req@ffff9d81984a0000 x1862023066594048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767888.053549:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066594048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767888.053557:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767888.053560:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767888.053563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767888.693082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767889.076967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767889.717008:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767890.100997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767890.741095:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767891.124953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767891.765024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767892.148977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767892.789064:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767893.172963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767893.172989:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767893.172995:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767893.172998:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767893.173006:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4724 00000100:00080000:3.0:1775767893.173015:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767893.173021:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767893.173054:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066594176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767893.173076:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767893.173079:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767893.173080:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767893.173083:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4724 00000100:00080000:3.0:1775767893.173085:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767893.173087:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767893.173095:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767893.173103:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066594304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767893.173111:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767893.173113:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767893.173115:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767893.173118:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4724 00000100:00080000:3.0:1775767893.173121:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767893.173124:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767893.173127:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767893.173134:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066594176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767893.173135:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066594432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767893.173148:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767893.173151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767893.173152:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767893.173154:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4724 00000100:00080000:3.0:1775767893.173156:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767893.173159:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767893.173171:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767893.173182:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066594560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767893.173189:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767893.173191:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767893.173192:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767893.173195:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4724 00000100:00080000:3.0:1775767893.173198:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767893.173202:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767893.173218:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767893.173222:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767893.173232:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066594688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767893.173270:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767893/real 1775767893] req@ffff9d8188740380 x1862023066594176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767893.173291:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066594176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767893.173302:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767893.173307:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767893.173311:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066594176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767893.173320:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767893.173344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767893.173358:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767893/real 1775767893] req@ffff9d8188741c00 x1862023066594304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767893.173371:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066594304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767893.173380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767893.173383:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767893.173387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767893.173410:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767893.173432:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767893.173444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767893/real 1775767893] req@ffff9d8188742a00 x1862023066594432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767893.173456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066594432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767893.173470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767893.173474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767893.173477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767893.173484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767893.173505:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767893.173528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767893/real 1775767893] req@ffff9d8188740700 x1862023066594560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767893.173541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066594560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767893.173553:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767893.173556:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767893.173559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767893.173565:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767893.173585:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767893.173596:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767893/real 1775767893] req@ffff9d8188740000 x1862023066594688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767893.173610:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066594688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767893.173620:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767893.173623:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767893.173627:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767893.813058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767894.196970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767894.837054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767895.221003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767895.860997:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767896.244975:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767896.885061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767897.268946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767897.909013:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767898.293030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767898.293038:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767898.293048:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767898.293062:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767898.293079:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4729 00000100:00080000:1.0:1775767898.293100:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767898.293110:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767898.293177:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066594816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767898.293211:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767898.293215:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767898.293217:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767898.293222:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4729 00000100:00080000:1.0:1775767898.293227:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767898.293232:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767898.293245:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767898.293253:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066594816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767898.293262:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066594944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767898.293289:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767898.293294:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767898.293296:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767898.293301:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4729 00000100:00080000:1.0:1775767898.293324:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767898.293329:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767898.293335:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775767898.293338:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767898.293354:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066595072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767898.293372:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767898.293376:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767898.293378:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767898.293383:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4729 00000100:00080000:1.0:1775767898.293387:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767898.293392:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767898.293397:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767898/real 1775767898] req@ffff9d8190553480 x1862023066594816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775767898.293400:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775767898.293418:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066595200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767898.293428:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066594816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767898.293430:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767898.293439:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767898.293444:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767898.293452:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4729 00000100:00080000:0.0:1775767898.293453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767898.293458:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767898.293462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:1.0:1775767898.293467:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767898.293472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066594816:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775767898.293484:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767898.293499:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066595328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767898.293503:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767898.293532:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767898.293544:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767898/real 1775767898] req@ffff9d8190553100 x1862023066594944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767898.293553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066594944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767898.293564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767898.293567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767898.293569:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066594944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767898.293573:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767898.293584:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767898.293590:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767898/real 1775767898] req@ffff9d8190551500 x1862023066595072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767898.293597:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066595072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767898.293603:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767898.293604:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767898.293606:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767898.293610:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767898.293621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767898.293626:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767898/real 1775767898] req@ffff9d8190551880 x1862023066595200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767898.293634:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066595200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767898.293640:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767898.293641:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767898.293644:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767898.293653:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767898.293664:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767898.293670:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767898/real 1775767898] req@ffff9d8190550e00 x1862023066595328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767898.293677:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066595328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767898.293683:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767898.293684:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767898.293686:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767898.933083:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767899.316948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767899.956992:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767900.340984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767900.981090:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767901.365001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767902.005055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767902.388996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767903.029070:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767903.412954:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767903.412956:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767903.412964:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767903.412968:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767903.412975:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4734 00000100:00080000:1.0:1775767903.412986:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767903.412991:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767903.413025:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066595456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767903.413051:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767903.413056:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767903.413058:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767903.413063:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4734 00000100:00080000:1.0:1775767903.413067:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767903.413072:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767903.413080:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066595456:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775767903.413083:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767903.413122:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066595584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767903.413133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767903.413137:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767903.413139:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767903.413143:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4734 00000100:00080000:1.0:1775767903.413148:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767903.413150:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775767903.413155:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767903.413161:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767903.413173:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066595712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767903.413183:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767903.413186:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767903.413188:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767903.413192:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4734 00000100:00080000:1.0:1775767903.413196:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767903.413201:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767903.413206:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767903.413220:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767903/real 1775767903] req@ffff9d8190550e00 x1862023066595456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767903.413220:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066595840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767903.413255:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767903.413259:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767903.413261:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767903.413264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066595456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767903.413269:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4734 00000100:00080000:1.0:1775767903.413276:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775767903.413280:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767903.413282:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775767903.413289:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000008:00100000:1.0:1775767903.413289:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767903.413295:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066595456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767903.413307:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066595968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767903.413324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767903.413351:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767903.413363:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767903/real 1775767903] req@ffff9d8190551880 x1862023066595584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767903.413375:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066595584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767903.413383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767903.413387:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767903.413390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767903.413396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767903.413411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767903.413420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767903/real 1775767903] req@ffff9d8190551500 x1862023066595712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767903.413431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066595712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767903.413439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767903.413441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767903.413445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767903.413450:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767903.413466:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767903.413474:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767903/real 1775767903] req@ffff9d8190553100 x1862023066595840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767903.413486:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066595840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767903.413496:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767903.413499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767903.413501:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767903.413506:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767903.413525:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767903.413535:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767903/real 1775767903] req@ffff9d8190553480 x1862023066595968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767903.413546:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066595968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767903.413554:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767903.413558:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767903.413560:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066595968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767904.053051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767904.436998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767905.077057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767905.460966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767906.101031:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767906.484968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767907.125000:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767907.508970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767908.149090:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767908.532970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767908.532988:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767908.532999:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767908.533003:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767908.533011:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4740 00000100:00080000:2.0:1775767908.533020:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767908.533025:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767908.533055:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066596096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767908.533087:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767908.533089:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767908.533091:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767908.533093:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4740 00000100:00080000:2.0:1775767908.533096:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767908.533099:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767908.533107:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767908.533119:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066596224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767908.533120:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066596096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767908.533133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767908.533135:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767908.533137:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767908.533141:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4740 00000100:00080000:2.0:1775767908.533143:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767908.533145:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767908.533149:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767908.533156:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066596352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767908.533162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767908.533165:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767908.533166:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767908.533168:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4740 00000100:00080000:2.0:1775767908.533177:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767908.533181:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767908.533186:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767908.533190:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767908.533205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066596480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767908.533216:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767908.533219:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767908.533221:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767908.533225:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4740 00000100:00080000:2.0:1775767908.533228:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767908.533233:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767908/real 1775767908] req@ffff9d81984a3b80 x1862023066596096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767908.533233:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767908.533240:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767908.533253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066596608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767908.533264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066596096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767908.533276:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767908.533281:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767908.533286:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066596096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767908.533308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767908.533327:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767908.533337:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767908/real 1775767908] req@ffff9d81984a3480 x1862023066596224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767908.533345:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066596224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767908.533351:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767908.533354:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767908.533356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767908.533369:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767908.533381:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767908.533388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767908/real 1775767908] req@ffff9d81984a2d80 x1862023066596352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767908.533396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066596352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767908.533401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767908.533403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767908.533405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767908.533409:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767908.533419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767908.533424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767908/real 1775767908] req@ffff9d81984a0700 x1862023066596480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767908.533431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066596480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767908.533438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767908.533440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767908.533441:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767908.533445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767908.533454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767908.533460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767908/real 1775767908] req@ffff9d81984a2a00 x1862023066596608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767908.533467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066596608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767908.533475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767908.533478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767908.533481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767909.173098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767909.556979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767910.197082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767910.580960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767911.221067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767911.604987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767912.245073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767912.629046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767913.269113:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767913.653003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767913.653028:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767913.653038:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767913.653042:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767913.653052:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4745 00000100:00080000:3.0:1775767913.653061:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767913.653068:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767913.653111:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066596736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767913.653131:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767913.653133:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767913.653135:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767913.653137:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4745 00000100:00080000:3.0:1775767913.653140:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767913.653142:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767913.653150:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767913.653158:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767913.653166:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767913.653172:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066596736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767913.653175:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767913.653178:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767913.653186:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4745 00000100:00080000:3.0:1775767913.653188:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767913.653191:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767913.653194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767913.653203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066596992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767913.653210:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767913.653212:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767913.653214:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767913.653217:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4745 00000100:00080000:3.0:1775767913.653230:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767913.653232:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767913.653236:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767913.653242:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767913.653246:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066597120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767913.653253:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767913.653256:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767913.653257:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767913.653260:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4745 00000100:00080000:3.0:1775767913.653262:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767913.653265:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767913.653268:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767913.653274:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066597248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767913.653302:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767913/real 1775767913] req@ffff9d8188741f80 x1862023066596736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767913.653325:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066596736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767913.653337:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767913.653341:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767913.653344:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066596736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767913.653363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767913.653382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767913.653391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767913/real 1775767913] req@ffff9d8188743800 x1862023066596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767913.653400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066596864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767913.653405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767913.653408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767913.653410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767913.653414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767913.653425:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767913.653430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767913/real 1775767913] req@ffff9d8188741880 x1862023066596992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767913.653438:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066596992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767913.653443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767913.653444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767913.653446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066596992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767913.653450:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767913.653469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767913.653476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767913/real 1775767913] req@ffff9d8188742680 x1862023066597120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767913.653488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066597120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767913.653495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767913.653497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767913.653499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767913.653503:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767913.653513:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767913.653518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767913/real 1775767913] req@ffff9d8188742300 x1862023066597248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767913.653525:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066597248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767913.653531:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767913.653532:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767913.653534:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767914.293098:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767914.677022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767915.317047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767915.701064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767916.341071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767916.724976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767917.365055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767917.748995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767918.389099:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767918.772979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767918.772994:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767918.773004:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767918.773009:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767918.773020:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4750 00000100:00080000:3.0:1775767918.773033:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767918.773040:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767918.773089:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066597376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767918.773120:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767918.773123:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767918.773124:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767918.773127:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4750 00000100:00080000:3.0:1775767918.773129:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767918.773132:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767918.773141:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767918.773151:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066597504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767918.773160:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767918.773168:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066597376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767918.773169:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767918.773176:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767918.773182:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4750 00000100:00080000:3.0:1775767918.773191:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767918.773194:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767918.773198:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767918.773208:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066597632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767918.773214:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767918.773217:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767918.773218:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767918.773222:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4750 00000100:00080000:3.0:1775767918.773224:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767918.773226:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767918.773229:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767918.773236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066597760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767918.773242:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767918.773246:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767918.773247:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767918.773249:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4750 00000100:00080000:3.0:1775767918.773252:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767918.773254:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767918.773256:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775767918.773263:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767918.773272:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066597888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767918.773325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767918/real 1775767918] req@ffff9d8188740e00 x1862023066597376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767918.773345:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066597376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767918.773354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767918.773358:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767918.773361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066597376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767918.773380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767918.773400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767918.773408:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767918/real 1775767918] req@ffff9d8188741180 x1862023066597504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767918.773417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066597504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767918.773422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767918.773425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767918.773427:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767918.773431:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767918.773442:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767918.773447:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767918/real 1775767918] req@ffff9d8188743100 x1862023066597632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767918.773460:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066597632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767918.773465:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767918.773467:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767918.773469:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767918.773473:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767918.773484:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767918.773489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767918/real 1775767918] req@ffff9d8188743480 x1862023066597760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767918.773497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066597760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767918.773503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767918.773505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767918.773507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767918.773510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767918.773521:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767918.773529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767918/real 1775767918] req@ffff9d8188743b80 x1862023066597888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767918.773541:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066597888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767918.773557:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767918.773561:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767918.773564:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066597888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767919.413035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767919.797046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767920.437084:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767920.821007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767921.461055:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767921.844987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767922.485070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767922.869033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767923.509128:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767923.893023:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767923.893034:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775767923.893038:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767923.893039:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767923.893051:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4755 00000100:00080000:3.0:1775767923.893071:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767923.893079:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767923.893129:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066598016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767923.893155:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767923.893158:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767923.893160:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767923.893163:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4755 00000100:00080000:3.0:1775767923.893165:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767923.893168:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767923.893178:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767923.893189:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767923.893200:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767923.893207:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775767923.893208:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066598016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767923.893209:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767923.893219:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4755 00000100:00080000:3.0:1775767923.893223:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767923.893225:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767923.893229:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767923.893245:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066598272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767923.893251:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767923.893254:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767923.893255:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767923.893258:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4755 00000100:00080000:3.0:1775767923.893279:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767923.893282:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767923.893288:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767923.893304:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066598400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767923.893315:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767923.893320:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767923.893323:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767923.893330:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4755 00000100:00080000:3.0:1775767923.893337:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767923.893342:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767923.893348:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767923.893357:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767923.893368:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066598528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767923.893402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767923.893430:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767923.893459:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767924.533040:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767924.916954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767925.557076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767925.940991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767926.581045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767926.964976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767926.965141:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767926.965185:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767926.965201:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767926.965239:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767926.965248:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767926.965258:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767926.965266:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767926.965293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767923/real 1775767926] req@ffff9d8188743480 x1862023066598528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767926.965328:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066598528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767926.965340:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767926.965346:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767926.965350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598528:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767926.965358:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767923/real 1775767926] req@ffff9d8188740a80 x1862023066598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767926.965372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066598144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767926.965381:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767926.965385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767926.965389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767926.965396:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767923/real 1775767926] req@ffff9d8188741500 x1862023066598272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767926.965407:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066598272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767926.965414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767926.965417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767926.965420:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767926.965426:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767923/real 1775767926] req@ffff9d8188743b80 x1862023066598400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767926.965440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066598400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767926.965450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767926.965454:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767926.965457:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767926.965464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767923/real 1775767926] req@ffff9d8188742d80 x1862023066598016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767926.965476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066598016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767926.965486:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767926.965489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767926.965492:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066598016:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775767927.605048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767927.988960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767928.629039:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767929.012980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767929.012991:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767929.012999:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767929.013004:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767929.013014:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4760 00000100:00080000:3.0:1775767929.013026:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767929.013033:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767929.013068:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066598656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767929.013097:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767929.013112:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767929.013114:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767929.013118:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4760 00000100:00080000:3.0:1775767929.013123:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767929.013128:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767929.013138:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767929.013147:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066598656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767929.013155:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066598784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767929.013168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767929.013172:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767929.013174:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767929.013180:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4760 00000100:00080000:3.0:1775767929.013195:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767929.013199:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767929.013206:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767929.013208:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767929.013225:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066598912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767929.013237:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767929.013240:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767929.013242:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767929.013248:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4760 00000100:00080000:3.0:1775767929.013252:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767929.013257:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767929/real 1775767929] req@ffff9d8188743100 x1862023066598656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767929.013261:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767929.013270:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775767929.013287:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066598656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767929.013287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066599040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767929.013324:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767929.013330:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767929.013336:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775767929.013344:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767929.013349:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767929.013351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066598656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767929.013358:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4760 00000100:00080000:3.0:1775767929.013364:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767929.013369:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767929.013374:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767929.013382:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767929.013392:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066599168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767929.013413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767929.013430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767929/real 1775767929] req@ffff9d8188741180 x1862023066598784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767929.013444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066598784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767929.013452:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767929.013456:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767929.013459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767929.013464:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767929.013484:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767929.013496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767929/real 1775767929] req@ffff9d8188740e00 x1862023066598912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767929.013508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066598912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767929.013516:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767929.013519:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767929.013523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066598912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767929.013529:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767929.013549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767929.013559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767929/real 1775767929] req@ffff9d8188742300 x1862023066599040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767929.013570:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066599040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767929.013579:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767929.013582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767929.013584:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767929.013599:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767929.013633:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767929.013645:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767929/real 1775767929] req@ffff9d8188742680 x1862023066599168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767929.013657:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066599168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767929.013665:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767929.013668:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767929.013671:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767929.653114:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767930.036952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767930.677043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767931.061049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767931.701115:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767932.084968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767932.725036:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767933.109020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767933.749072:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775767934.132974:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767934.132982:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775767934.132987:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767934.132996:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4765 00000100:00080000:1.0:1775767934.133005:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767934.133007:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775767934.133012:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775767934.133049:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066599296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767934.133075:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767934.133078:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775767934.133079:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767934.133083:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4765 00000100:00080000:1.0:1775767934.133088:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767934.133092:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767934.133100:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767934.133109:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066599296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767934.133112:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066599424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767934.133124:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767934.133126:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775767934.133127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767934.133130:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4765 00000100:00080000:1.0:1775767934.133132:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767934.133134:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767934.133137:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767934.133147:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066599552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767934.133154:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767934.133157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775767934.133158:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767934.133160:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4765 00000100:00080000:1.0:1775767934.133174:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767934.133177:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767934.133180:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767934.133187:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767934.133188:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066599680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775767934.133196:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775767934.133199:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775767934.133200:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775767934.133204:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4765 00000100:00080000:1.0:1775767934.133207:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775767934.133209:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775767934.133212:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767934.133220:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066599808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767934.133241:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767934/real 1775767934] req@ffff9d8190553480 x1862023066599296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767934.133263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066599296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767934.133275:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767934.133280:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767934.133283:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066599296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767934.133305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767934.133330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767934.133343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767934/real 1775767934] req@ffff9d8190553100 x1862023066599424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767934.133356:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066599424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767934.133365:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767934.133368:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767934.133372:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767934.133378:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767934.133400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767934.133412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767934/real 1775767934] req@ffff9d8190551500 x1862023066599552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767934.133426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066599552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767934.133435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767934.133438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767934.133442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767934.133447:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767934.133469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767934.133480:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767934/real 1775767934] req@ffff9d8190551880 x1862023066599680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767934.133493:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066599680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767934.133503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767934.133506:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767934.133509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767934.133515:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767934.133533:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767934.133542:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767934/real 1775767934] req@ffff9d8190550e00 x1862023066599808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767934.133553:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066599808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767934.133562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767934.133566:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767934.133569:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066599808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767934.773025:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767935.157036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767935.797060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767936.180997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767936.821029:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767937.205066:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767937.845096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767938.229105:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767938.869033:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767939.253019:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767939.253027:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767939.253031:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767939.253040:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767939.253041:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4770 00000100:00080000:2.0:1775767939.253053:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767939.253061:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767939.253097:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066599936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767939.253120:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767939.253123:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767939.253124:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767939.253126:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4770 00000100:00080000:2.0:1775767939.253129:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767939.253131:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767939.253139:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767939.253149:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066600064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767939.253158:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767939.253160:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767939.253162:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767939.253167:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066599936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775767939.253170:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4770 00000100:00080000:2.0:1775767939.253189:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767939.253196:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767939.253203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767939.253218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066600192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767939.253229:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767939.253232:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767939.253234:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767939.253237:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4770 00000100:00080000:2.0:1775767939.253251:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767939.253254:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767939.253259:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767939.253266:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767939.253271:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066600320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767939.253278:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767939.253280:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767939.253281:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767939.253283:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4770 00000100:00080000:2.0:1775767939.253285:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767939.253289:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767939.253293:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767939.253304:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066600448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767939.253342:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767939/real 1775767939] req@ffff9d81984a2300 x1862023066599936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767939.253360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066599936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767939.253369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767939.253374:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767939.253379:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066599936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767939.253408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767939.253438:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767939.253452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767939/real 1775767939] req@ffff9d81984a2a00 x1862023066600064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767939.253461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066600064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767939.253467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767939.253470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767939.253472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767939.253477:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767939.253493:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767939.253499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767939/real 1775767939] req@ffff9d81984a0700 x1862023066600192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767939.253507:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066600192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767939.253512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767939.253514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767939.253516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767939.253520:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767939.253532:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767939.253538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767939/real 1775767939] req@ffff9d81984a2d80 x1862023066600320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767939.253558:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066600320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767939.253568:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767939.253571:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767939.253575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767939.253582:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767939.253604:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767939.253613:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767939/real 1775767939] req@ffff9d81984a3480 x1862023066600448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767939.253621:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066600448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767939.253628:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767939.253630:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767939.253632:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775767939.893057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767940.276965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767940.917080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767941.301023:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767941.941105:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767942.324978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775767942.965094:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767943.349056:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767943.989076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767944.372967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767944.372970:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767944.372978:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767944.372982:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767944.372990:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4775 00000100:00080000:2.0:1775767944.372999:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767944.373006:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767944.373042:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066600576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767944.373072:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767944.373077:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767944.373079:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767944.373085:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4775 00000100:00080000:2.0:1775767944.373090:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767944.373095:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767944.373105:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767944.373111:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066600576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767944.373121:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066600704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767944.373130:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767944.373133:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767944.373134:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767944.373137:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4775 00000100:00080000:2.0:1775767944.373140:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767944.373143:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767944.373146:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767944.373154:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066600832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767944.373165:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767944.373167:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767944.373169:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767944.373172:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4775 00000100:00080000:2.0:1775767944.373174:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767944.373176:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767944.373179:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767944.373183:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767944.373191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066600960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767944.373198:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767944.373201:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767944.373202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767944.373205:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4775 00000100:00080000:2.0:1775767944.373207:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767944.373209:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767944.373212:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767944.373223:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066601088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767944.373240:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767944/real 1775767944] req@ffff9d81984a3b80 x1862023066600576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767944.373262:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066600576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767944.373273:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767944.373277:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767944.373281:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066600576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767944.373305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767944.373340:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767944.373354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767944/real 1775767944] req@ffff9d81984a0000 x1862023066600704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767944.373368:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066600704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767944.373376:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767944.373380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767944.373382:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767944.373387:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767944.373403:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767944.373410:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767944/real 1775767944] req@ffff9d81984a2680 x1862023066600832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767944.373422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066600832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767944.373428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767944.373430:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767944.373432:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767944.373435:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767944.373446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767944.373451:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767944/real 1775767944] req@ffff9d81984a0e00 x1862023066600960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767944.373458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066600960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767944.373464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767944.373466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767944.373468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066600960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767944.373471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767944.373480:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767944.373485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767944/real 1775767944] req@ffff9d81984a1f80 x1862023066601088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767944.373492:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066601088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767944.373502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767944.373504:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767944.373506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767945.013002:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767945.396970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767946.037033:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767946.420969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767947.061068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767947.444943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767948.085068:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767948.469084:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767949.109048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767949.492951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767949.492962:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767949.492967:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767949.492971:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767949.492979:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4781 00000100:00080000:2.0:1775767949.492988:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767949.492994:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767949.493028:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066601216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767949.493057:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767949.493061:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767949.493063:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767949.493069:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4781 00000100:00080000:2.0:1775767949.493074:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767949.493078:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767949.493084:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066601216:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775767949.493088:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767949.493104:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066601344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767949.493112:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767949.493115:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767949.493117:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767949.493120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4781 00000100:00080000:2.0:1775767949.493130:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767949.493133:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767949.493138:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775767949.493139:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767949.493150:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066601472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767949.493156:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767949.493160:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767949.493161:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767949.493163:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4781 00000100:00080000:2.0:1775767949.493166:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767949.493169:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767949.493173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767949.493179:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066601600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767949.493185:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767949/real 1775767949] req@ffff9d81984a1180 x1862023066601216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767949.493188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767949.493195:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767949.493198:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767949.493206:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4781 00000100:00080000:2.0:1775767949.493209:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767949.493212:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767949.493214:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767949.493215:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066601216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775767949.493224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066601728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767949.493226:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767949.493235:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767949.493239:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066601216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767949.493257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767949.493272:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767949.493280:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767949/real 1775767949] req@ffff9d81984a1880 x1862023066601344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767949.493288:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066601344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767949.493294:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767949.493297:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767949.493299:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767949.493302:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767949.493312:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767949.493317:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767949/real 1775767949] req@ffff9d81984a3800 x1862023066601472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767949.493324:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066601472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767949.493329:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767949.493331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767949.493333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767949.493336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767949.493346:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767949.493351:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767949/real 1775767949] req@ffff9d81984a3100 x1862023066601600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767949.493358:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066601600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767949.493365:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767949.493366:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767949.493369:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767949.493373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767949.493382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767949.493387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767949/real 1775767949] req@ffff9d81984a1500 x1862023066601728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767949.493393:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066601728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767949.493398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767949.493400:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767949.493402:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767950.133037:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767950.517012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767951.157085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767951.540940:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767952.181032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767952.564990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767953.205052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767953.588965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767954.229039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767954.612979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767954.613025:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767954.613050:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767954.613056:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767954.613068:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4786 00000100:00080000:3.0:1775767954.613083:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767954.613091:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767954.613142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066601856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767954.613171:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767954.613173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767954.613175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767954.613179:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4786 00000100:00080000:3.0:1775767954.613182:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767954.613186:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767954.613196:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767954.613206:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066601984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767954.613213:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066601856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767954.613233:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767954.613236:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767954.613237:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767954.613240:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4786 00000100:00080000:3.0:1775767954.613243:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767954.613246:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767954.613251:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767954.613266:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066602112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767954.613272:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767954.613275:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767954.613277:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767954.613280:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4786 00000100:00080000:3.0:1775767954.613289:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767954.613292:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767954.613296:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767954.613302:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775767954.613316:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066602240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767954.613325:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767954.613328:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767954.613329:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767954.613333:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4786 00000100:00080000:3.0:1775767954.613337:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767954.613340:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767954.613345:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767954.613365:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066602368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767954.613375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767954/real 1775767954] req@ffff9d8188741880 x1862023066601856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767954.613395:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066601856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767954.613410:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767954.613413:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767954.613417:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066601856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767954.613436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767954.613454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767954.613462:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767954/real 1775767954] req@ffff9d8188743800 x1862023066601984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767954.613470:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066601984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767954.613476:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767954.613478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767954.613480:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066601984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767954.613484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767954.613494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767954.613499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767954/real 1775767954] req@ffff9d8188741f80 x1862023066602112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767954.613527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066602112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767954.613534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767954.613537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767954.613539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767954.613545:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767954.613560:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767954.613583:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767954/real 1775767954] req@ffff9d8188740000 x1862023066602240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767954.613591:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066602240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767954.613598:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767954.613600:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767954.613601:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767954.613605:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767954.613646:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767954.613654:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767954/real 1775767954] req@ffff9d8188740700 x1862023066602368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767954.613661:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066602368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767954.613667:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767954.613669:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767954.613671:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767955.253076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767955.636955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767956.277037:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767956.661027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767957.301077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767957.684985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767958.325060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767958.708986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767959.349068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767959.733000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767959.733032:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767959.733038:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767959.733041:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767959.733049:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4791 00000100:00080000:2.0:1775767959.733057:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767959.733062:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767959.733093:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066602496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767959.733116:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767959.733119:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767959.733120:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767959.733123:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4791 00000100:00080000:2.0:1775767959.733125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767959.733128:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767959.733135:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767959.733146:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066602496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767959.733149:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066602624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767959.733159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767959.733161:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767959.733162:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767959.733166:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4791 00000100:00080000:2.0:1775767959.733168:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767959.733170:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767959.733173:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767959.733181:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066602752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767959.733186:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767959.733188:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767959.733189:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767959.733191:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4791 00000100:00080000:2.0:1775767959.733195:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767959.733197:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767959.733200:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775767959.733205:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775767959.733212:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066602880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767959.733219:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767959.733221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767959.733222:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767959.733224:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4791 00000100:00080000:2.0:1775767959.733226:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767959.733229:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767959.733232:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767959.733239:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066603008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767959.733246:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767959/real 1775767959] req@ffff9d81984a0a80 x1862023066602496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767959.733263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066602496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767959.733271:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767959.733274:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767959.733277:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066602496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767959.733294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767959.733309:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767959.733317:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767959/real 1775767959] req@ffff9d81984a1500 x1862023066602624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767959.733325:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066602624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767959.733330:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767959.733333:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767959.733335:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767959.733338:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767959.733358:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767959.733365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767959/real 1775767959] req@ffff9d81984a3100 x1862023066602752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767959.733372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066602752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767959.733382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767959.733384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767959.733386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767959.733390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767959.733400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767959.733405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767959/real 1775767959] req@ffff9d81984a3800 x1862023066602880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767959.733412:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066602880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767959.733424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767959.733426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767959.733428:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066602880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767959.733432:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767959.733443:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767959.733449:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767959/real 1775767959] req@ffff9d81984a1880 x1862023066603008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767959.733456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066603008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767959.733461:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767959.733463:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767959.733466:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767960.373010:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767960.756976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767961.397023:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767961.780952:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767962.421048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767962.804984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767963.445044:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767963.828947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767964.469039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775767964.852968:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767964.852972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767964.852978:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767964.852982:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767964.852991:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4796 00000100:00080000:2.0:1775767964.853000:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767964.853008:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767964.853047:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066603136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767964.853070:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767964.853076:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767964.853078:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767964.853082:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4796 00000100:00080000:2.0:1775767964.853087:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767964.853091:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767964.853100:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767964.853102:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066603136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775767964.853112:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066603264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767964.853123:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767964.853126:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767964.853127:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767964.853131:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4796 00000100:00080000:2.0:1775767964.853134:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767964.853137:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767964.853141:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767964.853151:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066603392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767964.853159:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775767964.853163:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775767964.853163:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767964.853164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767964.853168:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4796 00000100:00080000:2.0:1775767964.853171:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767964.853174:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767964.853179:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767964.853189:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066603520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767964.853199:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767964.853202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767964.853204:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767964.853207:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4796 00000100:00080000:2.0:1775767964.853211:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767964.853213:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767964/real 1775767964] req@ffff9d81984a1180 x1862023066603136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775767964.853214:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767964.853219:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775767964.853236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066603648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767964.853237:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066603136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767964.853250:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767964.853254:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767964.853258:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066603136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767964.853281:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767964.853303:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767964.853312:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767964/real 1775767964] req@ffff9d81984a1f80 x1862023066603264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767964.853321:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066603264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767964.853327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767964.853329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767964.853332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767964.853336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767964.853348:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767964.853353:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767964/real 1775767964] req@ffff9d81984a0e00 x1862023066603392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767964.853361:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066603392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767964.853366:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767964.853368:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767964.853369:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767964.853373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767964.853383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767964.853388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767964/real 1775767964] req@ffff9d81984a2680 x1862023066603520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767964.853395:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066603520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767964.853401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767964.853403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767964.853405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767964.853408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767964.853417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767964.853422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767964/real 1775767964] req@ffff9d81984a0000 x1862023066603648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767964.853429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066603648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767964.853435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767964.853436:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767964.853438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767965.493005:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767965.876931:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767966.517085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767966.900986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767967.541068:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767967.925001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767968.565068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767968.948969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767969.589040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767969.972983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767969.972988:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767969.972994:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767969.972998:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767969.973005:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4801 00000100:00080000:3.0:1775767969.973014:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767969.973020:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767969.973052:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066603776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767969.973075:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767969.973078:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767969.973079:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767969.973082:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4801 00000100:00080000:3.0:1775767969.973084:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767969.973086:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767969.973094:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767969.973102:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066603904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767969.973110:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767969.973112:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767969.973113:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767969.973116:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4801 00000100:00080000:3.0:1775767969.973119:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775767969.973121:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066603776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767969.973123:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767969.973130:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767969.973138:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066604032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767969.973149:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767969.973152:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767969.973153:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767969.973156:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4801 00000100:00080000:3.0:1775767969.973158:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767969.973160:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767969.973163:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767969.973171:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066604160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767969.973177:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767969.973179:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767969.973180:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767969.973182:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4801 00000100:00080000:3.0:1775767969.973185:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775767969.973189:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767969.973191:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767969.973195:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767969.973202:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066604288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767969.973232:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767969/real 1775767969] req@ffff9d8188742a00 x1862023066603776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767969.973246:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066603776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767969.973254:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767969.973257:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767969.973260:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066603776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767969.973265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767969.973279:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767969.973287:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767969/real 1775767969] req@ffff9d8188741c00 x1862023066603904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767969.973295:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066603904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767969.973300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767969.973302:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767969.973304:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066603904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767969.973324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767969.973343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767969.973365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767969/real 1775767969] req@ffff9d8188740380 x1862023066604032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767969.973380:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066604032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767969.973389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767969.973392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767969.973395:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767969.973402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767969.973423:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767969.973434:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767969/real 1775767969] req@ffff9d8188740700 x1862023066604160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767969.973447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066604160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767969.973458:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767969.973460:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767969.973477:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767969.973482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767969.973498:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767969.973505:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767969/real 1775767969] req@ffff9d8188740000 x1862023066604288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767969.973513:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066604288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775767994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767969.973519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767969.973522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767969.973523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767970.613035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767970.996976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767971.637008:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767972.020955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767972.661030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767973.044958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767973.685037:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767974.069024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767974.709063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767975.092980:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767975.092993:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767975.092999:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767975.093012:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4806 00000100:00080000:3.0:1775767975.093026:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767975.093035:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767975.093086:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066604416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767975.093131:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767975.093135:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767975.093138:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767975.093143:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4806 00000100:00080000:3.0:1775767975.093146:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767975.093151:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767975.093163:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775767975.093170:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066604416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767975.093176:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066604544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767975.093186:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767975.093189:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767975.093191:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767975.093204:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4806 00000100:00080000:3.0:1775767975.093211:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767975.093215:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767975.093220:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767975.093231:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066604672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767975.093240:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767975.093244:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767975.093245:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775767975.093247:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767975.093250:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4806 00000100:00080000:3.0:1775767975.093253:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767975.093256:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767975.093261:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767975.093271:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066604800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767975.093280:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767975.093283:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767975.093284:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767975.093287:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4806 00000100:00080000:3.0:1775767975.093290:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767975.093294:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767975.093298:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775767975.093306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767975/real 1775767975] req@ffff9d8188741f80 x1862023066604416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767975.093306:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066604928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775767975.093328:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066604416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767975.093339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767975.093344:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775767975.093349:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066604416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775767975.093377:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775767975.093397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767975.093406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767975/real 1775767975] req@ffff9d8188743800 x1862023066604544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767975.093429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066604544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767975.093437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767975.093439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775767975.093442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775767975.093446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775767975.093462:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767975.093470:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767975/real 1775767975] req@ffff9d8188741880 x1862023066604672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767975.093478:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066604672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767975.093488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767975.093491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775767975.093493:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775767975.093496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775767975.093507:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767975.093512:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767975/real 1775767975] req@ffff9d8188742680 x1862023066604800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767975.093519:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066604800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767975.093526:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767975.093527:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775767975.093529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775767975.093533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775767975.093544:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775767975.093549:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767975/real 1775767975] req@ffff9d8188742300 x1862023066604928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775767975.093556:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066604928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775767975.093562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775767975.093564:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775767975.093566:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066604928:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767975.094484:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767975.733025:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767976.116993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767976.757054:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767977.141008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767977.781007:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767978.164999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767978.805078:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767979.188994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767979.829028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767980.212979:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775767980.212981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767980.212990:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767980.212995:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767980.213006:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4811 00000100:00080000:3.0:1775767980.213017:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767980.213025:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767980.213076:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066605056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767980.213108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767980.213113:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767980.213115:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767980.213121:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4811 00000100:00080000:3.0:1775767980.213127:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767980.213132:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767980.213142:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066605056:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775767980.213143:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767980.213162:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066605184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767980.213174:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767980.213178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767980.213180:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767980.213185:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4811 00000100:00080000:3.0:1775767980.213200:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767980.213204:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767980.213210:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767980.213211:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767980.213228:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066605312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767980.213238:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767980.213242:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767980.213244:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767980.213250:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4811 00000100:00080000:3.0:1775767980.213255:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767980.213259:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767980.213264:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767980/real 1775767980] req@ffff9d8188740e00 x1862023066605056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767980.213267:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767980.213286:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066605440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767980.213292:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066605056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767980.213305:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775767980.213311:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775767980.213311:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767980.213314:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775767980.213318:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775767980.213323:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4811 00000100:00100000:0.0:1775767980.213324:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066605056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767980.213333:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767980.213340:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767980.213346:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767980.213350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775767980.213361:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066605568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775767980.213398:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767980.213412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767980/real 1775767980] req@ffff9d8188741180 x1862023066605184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767980.213422:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066605184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767980.213428:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767980.213431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767980.213433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767980.213437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767980.213450:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767980.213456:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767980/real 1775767980] req@ffff9d8188743100 x1862023066605312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767980.213467:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066605312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767980.213473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767980.213474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767980.213476:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767980.213480:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767980.213490:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767980.213496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767980/real 1775767980] req@ffff9d8188742d80 x1862023066605440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767980.213503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066605440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767980.213510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767980.213511:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767980.213513:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767980.213523:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767980.213533:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767980.213538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767980/real 1775767980] req@ffff9d8188743b80 x1862023066605568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767980.213545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066605568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767980.213555:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767980.213557:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767980.213559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767980.853064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767981.236973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767981.877043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767982.260997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767982.901062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767983.285026:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767983.925058:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767984.309017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767984.949046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775767985.332948:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767985.332955:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767985.332959:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767985.332961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767985.332968:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4816 00000100:00080000:3.0:1775767985.332977:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767985.332983:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767985.333015:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066605696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767985.333038:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767985.333041:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767985.333042:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767985.333045:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4816 00000100:00080000:3.0:1775767985.333048:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767985.333051:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767985.333059:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775767985.333068:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066605696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775767985.333070:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767985.333081:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767985.333083:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767985.333084:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767985.333087:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4816 00000100:00080000:3.0:1775767985.333089:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767985.333092:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767985.333095:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767985.333105:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066605952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767985.333115:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767985.333119:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767985.333121:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767985.333124:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4816 00000800:00000100:0.0:1775767985.333128:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775767985.333132:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767985.333138:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767985.333145:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767985.333160:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066606080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767985.333170:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767985.333172:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767985.333173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767985.333178:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767985/real 1775767985] req@ffff9d8188741500 x1862023066605696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775767985.333179:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4816 00000100:00080000:3.0:1775767985.333186:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767985.333192:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767985.333197:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767985.333202:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066605696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775767985.333208:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066606208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775767985.333216:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767985.333222:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767985.333226:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066605696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767985.333245:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767985.333260:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767985.333269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767985/real 1775767985] req@ffff9d8188740a80 x1862023066605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767985.333282:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066605824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767985.333291:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767985.333295:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767985.333298:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767985.333305:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767985.333357:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767985.333366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767985/real 1775767985] req@ffff9d8188743480 x1862023066605952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767985.333374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066605952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767985.333380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767985.333382:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767985.333384:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066605952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767985.333388:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767985.333400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767985.333406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767985/real 1775767985] req@ffff9d8188743b80 x1862023066606080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767985.333414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066606080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767985.333420:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767985.333422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767985.333424:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767985.333428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767985.333438:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767985.333443:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767985/real 1775767985] req@ffff9d8188742d80 x1862023066606208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767985.333450:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066606208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767985.333455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767985.333457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767985.333459:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767985.973125:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767986.356978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767986.997073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767987.380986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767988.021031:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767988.405048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767989.045072:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775767989.428989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767990.069071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767990.452982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775767990.452986:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767990.452992:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775767990.452997:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767990.453005:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4821 00000100:00080000:2.0:1775767990.453014:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767990.453020:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775767990.453050:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066606336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767990.453073:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767990.453077:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775767990.453079:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767990.453083:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4821 00000100:00080000:2.0:1775767990.453088:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767990.453093:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767990.453100:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066606336:192.168.202.105@tcp:250:kworker.0 00000008:00100000:2.0:1775767990.453103:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767990.453120:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066606464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767990.453131:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767990.453135:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775767990.453137:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767990.453140:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4821 00000100:00080000:2.0:1775767990.453146:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767990.453149:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767990.453153:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775767990.453162:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066606592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775767990.453168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767990.453170:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775767990.453172:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767990.453174:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4821 00000100:00080000:2.0:1775767990.453176:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767990.453179:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767990.453183:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775767990.453191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066606720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767990.453201:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606464:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775767990.453215:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775767990.453218:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775767990.453220:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775767990.453224:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4821 00000100:00080000:2.0:1775767990.453226:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775767990.453229:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775767990.453233:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775767990.453237:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775767990.453247:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066606848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767990.453285:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767990.453310:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767991.093059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767991.476995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767992.117045:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767992.500954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767993.141074:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767993.524997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775767993.525201:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775767993.525265:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775767993.525282:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767993.525319:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767993.525328:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767993.525336:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775767993.525344:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775767993.525379:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767990/real 1775767993] req@ffff9d81984a0700 x1862023066606720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767993.525413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066606720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767993.525429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767993.525434:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767993.525438:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767993.525445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767990/real 1775767993] req@ffff9d81984a2a00 x1862023066606848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767993.525459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066606848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767993.525467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767993.525471:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767993.525475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606848:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767993.525482:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767990/real 1775767993] req@ffff9d81984a3480 x1862023066606464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767993.525493:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066606464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767993.525501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767993.525505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767993.525508:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767993.525514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767990/real 1775767993] req@ffff9d81984a2d80 x1862023066606592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767993.525527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066606592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767993.525535:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767993.525538:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767993.525541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066606592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767993.525548:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767990/real 1775767993] req@ffff9d81984a3b80 x1862023066606336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767993.525562:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066606336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767993.525569:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767993.525572:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767993.525575:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066606336:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775767994.165046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1775767994.549004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767995.189035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767995.572993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775767995.572994:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767995.573001:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775767995.573004:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767995.573012:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4827 00000100:00080000:3.0:1775767995.573021:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767995.573027:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775767995.573071:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066606976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767995.573101:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767995.573104:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775767995.573106:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767995.573109:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4827 00000100:00080000:3.0:1775767995.573113:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767995.573118:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767995.573127:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767995.573141:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767995.573153:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767995.573156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775767995.573159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775767995.573162:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066606976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775767995.573167:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4827 00000100:00080000:3.0:1775767995.573175:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767995.573179:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775767995.573186:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775767995.573200:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066607232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767995.573209:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767995.573213:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775767995.573231:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767995.573236:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4827 00000100:00080000:3.0:1775767995.573243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767995.573246:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775767995.573253:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775767995.573253:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775767995.573271:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066607360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775767995.573282:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775767995.573286:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775767995.573288:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775767995.573293:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4827 00000100:00080000:3.0:1775767995.573298:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775767995.573302:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775767995.573307:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767995/real 1775767995] req@ffff9d8188743100 x1862023066606976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775767995.573308:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775767995.573327:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066607488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775767995.573343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066606976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767995.573357:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767995.573361:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767995.573365:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066606976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775767995.573388:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767995.573412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767995.573424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767995/real 1775767995] req@ffff9d8188741180 x1862023066607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767995.573436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066607104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767995.573444:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767995.573448:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767995.573451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767995.573457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775767995.573477:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767995.573486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767995/real 1775767995] req@ffff9d8188740e00 x1862023066607232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767995.573497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066607232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767995.573505:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767995.573508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767995.573511:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775767995.573517:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767995.573554:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767995.573567:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767995/real 1775767995] req@ffff9d8188742300 x1862023066607360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767995.573579:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066607360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767995.573590:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767995.573593:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767995.573596:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775767995.573602:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775767995.573622:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775767995.573633:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775767995/real 1775767995] req@ffff9d8188742680 x1862023066607488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775767995.573644:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066607488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775767995.573652:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775767995.573655:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775767995.573658:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775767996.213089:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767996.597001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767997.237022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767997.620972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767998.261048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767998.644987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775767999.285006:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775767999.668993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768000.309059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768000.692993:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768000.692998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768000.693006:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768000.693012:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768000.693031:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4832 00000100:00080000:3.0:1775768000.693046:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768000.693054:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768000.693110:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066607616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768000.693142:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768000.693146:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768000.693148:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768000.693152:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4832 00000100:00080000:3.0:1775768000.693155:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768000.693160:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768000.693192:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768000.693207:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066607744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768000.693218:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066607616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768000.693221:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768000.693227:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768000.693229:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768000.693234:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4832 00000100:00080000:3.0:1775768000.693239:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768000.693243:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768000.693249:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768000.693263:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066607872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768000.693273:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768000.693277:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768000.693279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768000.693284:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4832 00000100:00080000:3.0:1775768000.693288:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768000.693292:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768000.693297:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775768000.693299:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768000.693313:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066608000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768000.693324:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768000.693328:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768000.693336:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768000.693340:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4832 00000100:00080000:3.0:1775768000.693344:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768000.693349:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768000.693354:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768000.693360:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768000/real 1775768000] req@ffff9d8188741880 x1862023066607616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768000.693367:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066608128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768000.693401:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066607616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768000.693415:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768000.693421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768000.693426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066607616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768000.693455:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768000.693481:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768000.693494:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768000/real 1775768000] req@ffff9d8188743800 x1862023066607744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768000.693509:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066607744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768000.693519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768000.693522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768000.693526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768000.693533:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768000.693550:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768000.693565:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768000/real 1775768000] req@ffff9d8188741f80 x1862023066607872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768000.693579:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066607872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768000.693588:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768000.693592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768000.693596:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066607872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768000.693602:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768000.693621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768000.693630:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768000/real 1775768000] req@ffff9d8188740000 x1862023066608000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768000.693639:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066608000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768000.693646:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768000.693648:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768000.693650:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768000.693655:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768000.693668:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768000.693674:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768000/real 1775768000] req@ffff9d8188740700 x1862023066608128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768000.693681:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066608128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768000.693687:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768000.693689:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768000.693690:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768001.333132:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768001.717054:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768002.357043:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768002.740974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768003.381075:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768003.765024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768004.405049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768004.788939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768005.429093:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768005.812969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768005.813023:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768005.813034:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768005.813040:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768005.813053:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4837 00000100:00080000:2.0:1775768005.813067:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768005.813076:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768005.813133:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066608256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768005.813168:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768005.813173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768005.813174:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768005.813178:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4837 00000100:00080000:2.0:1775768005.813183:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768005.813187:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768005.813199:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768005.813203:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066608256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768005.813217:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066608384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768005.813230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768005.813243:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768005.813246:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768005.813252:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4837 00000100:00080000:2.0:1775768005.813259:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768005.813264:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768005.813273:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775768005.813275:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768005.813294:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066608512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768005.813306:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768005.813310:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768005.813313:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768005.813319:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4837 00000100:00080000:2.0:1775768005.813325:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768005.813330:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768005/real 1775768005] req@ffff9d81984a2300 x1862023066608256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768005.813333:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768005.813344:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768005.813358:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066608256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768005.813364:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066608640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768005.813369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768005.813375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768005.813384:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066608256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768005.813409:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768005.813412:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768005.813414:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768005.813419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608384:192.168.202.105@tcp:38:kworker.0 00000100:00080000:2.0:1775768005.813422:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4837 00000100:00080000:2.0:1775768005.813435:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768005.813441:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768005.813450:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775768005.813452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768005.813466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768005/real 1775768005] req@ffff9d81984a3b80 x1862023066608384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768005.813475:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066608768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768005.813484:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066608384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768005.813514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768005.813518:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768005.813521:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768005.813526:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768005.813544:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768005.813551:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768005/real 1775768005] req@ffff9d81984a2d80 x1862023066608512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768005.813558:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066608512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768005.813562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768005.813564:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768005.813565:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768005.813569:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768005.813579:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768005.813584:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768005/real 1775768005] req@ffff9d81984a3480 x1862023066608640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768005.813593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066608640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768005.813602:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768005.813605:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768005.813607:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768005.813622:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768005.813649:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768005.813659:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768005/real 1775768005] req@ffff9d81984a2a00 x1862023066608768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768005.813671:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066608768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768005.813676:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768005.813678:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768005.813680:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066608768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768006.453089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768006.837024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768007.477125:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768007.860944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768008.501124:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768008.884983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768009.525130:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768009.908970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768010.549063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768010.932993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768010.932993:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768010.933003:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768010.933008:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768010.933018:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4842 00000100:00080000:3.0:1775768010.933032:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768010.933039:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768010.933104:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066608896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768010.933128:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768010.933132:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768010.933133:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768010.933137:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4842 00000100:00080000:3.0:1775768010.933140:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768010.933144:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768010.933151:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066608896:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768010.933154:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768010.933164:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066609024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768010.933172:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768010.933175:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768010.933177:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768010.933181:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4842 00000100:00080000:3.0:1775768010.933185:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768010.933189:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768010.933194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768010.933205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066609152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768010.933214:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768010.933218:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768010.933219:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768010.933223:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4842 00000800:00000100:0.0:1775768010.933224:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768010.933232:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768010.933236:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768010.933241:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768010.933251:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066609280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768010.933257:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768010.933259:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768010.933261:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768010.933263:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4842 00000100:00080000:3.0:1775768010.933266:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768010.933268:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768010.933271:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768010.933275:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768010/real 1775768010] req@ffff9d8188740380 x1862023066608896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768010.933279:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066609408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768010.933307:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066608896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768010.933315:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768010.933319:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768010.933322:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066608896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768010.933341:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768010.933356:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768010.933364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768010/real 1775768010] req@ffff9d8188741c00 x1862023066609024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768010.933372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066609024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768010.933377:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768010.933379:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768010.933381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768010.933384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768010.933393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768010.933397:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768010/real 1775768010] req@ffff9d8188742a00 x1862023066609152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768010.933404:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066609152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768010.933409:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768010.933410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768010.933412:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768010.933414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768010.933424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768010.933429:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768010/real 1775768010] req@ffff9d8188740700 x1862023066609280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768010.933435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066609280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768010.933441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768010.933443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768010.933445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768010.933448:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768010.933456:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768010.933460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768010/real 1775768010] req@ffff9d8188740000 x1862023066609408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768010.933466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066609408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768010.933471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768010.933472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768010.933474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768011.573110:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768011.956969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768012.597028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768012.980993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768013.621027:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768014.004927:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768014.645156:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768015.028939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768015.669059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768016.052950:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768016.052957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768016.052957:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768016.052961:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768016.052969:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4847 00000100:00080000:3.0:1775768016.052977:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768016.052982:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768016.053030:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066609536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768016.053049:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768016.053052:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768016.053053:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768016.053055:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4847 00000100:00080000:3.0:1775768016.053058:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768016.053060:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768016.053067:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768016.053077:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066609664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768016.053080:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066609536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768016.053091:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768016.053093:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768016.053094:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768016.053097:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4847 00000100:00080000:3.0:1775768016.053099:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768016.053101:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768016.053103:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768016.053112:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066609792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768016.053117:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768016.053119:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768016.053121:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768016.053123:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4847 00000100:00080000:3.0:1775768016.053125:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768016.053127:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768016.053131:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768016.053133:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768016.053142:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066609920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768016.053148:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768016.053150:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768016.053151:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768016.053153:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4847 00000100:00080000:3.0:1775768016.053155:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768016.053157:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768016.053160:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768016.053168:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066610048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768016.053171:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768016/real 1775768016] req@ffff9d8188741f80 x1862023066609536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768016.053192:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066609536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768016.053199:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768016.053202:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768016.053205:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066609536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768016.053220:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768016.053233:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768016.053239:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768016/real 1775768016] req@ffff9d8188743800 x1862023066609664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768016.053246:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066609664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768016.053251:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768016.053253:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768016.053254:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768016.053257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768016.053265:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768016.053270:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768016/real 1775768016] req@ffff9d8188741880 x1862023066609792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768016.053276:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066609792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768016.053284:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768016.053286:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768016.053288:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768016.053291:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768016.053299:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768016.053303:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768016/real 1775768016] req@ffff9d8188742680 x1862023066609920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768016.053309:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066609920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768016.053315:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768016.053317:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768016.053318:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066609920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768016.053321:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768016.053329:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768016.053333:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768016/real 1775768016] req@ffff9d8188742300 x1862023066610048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768016.053340:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066610048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768016.053347:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768016.053350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768016.053353:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768016.692994:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768017.077056:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768017.717055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768018.101079:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768018.741126:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768019.125046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768019.765098:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768020.149045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768020.789100:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768021.173019:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768021.173028:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768021.173032:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768021.173036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768021.173041:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4852 00000100:00080000:3.0:1775768021.173051:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768021.173057:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768021.173090:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066610176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768021.173118:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768021.173121:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768021.173122:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768021.173125:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4852 00000100:00080000:3.0:1775768021.173127:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768021.173131:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768021.173139:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768021.173148:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066610304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768021.173155:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066610176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768021.173158:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768021.173162:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768021.173163:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768021.173166:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4852 00000100:00080000:3.0:1775768021.173168:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768021.173170:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768021.173174:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768021.173181:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066610432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768021.173187:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768021.173190:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768021.173191:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768021.173194:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4852 00000100:00080000:3.0:1775768021.173202:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768021.173204:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768021.173209:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775768021.173209:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768021.173219:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066610560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768021.173226:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768021.173228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768021.173230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768021.173232:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4852 00000100:00080000:3.0:1775768021.173234:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768021.173237:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768021.173240:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768021.173249:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066610688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768021.173252:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768021/real 1775768021] req@ffff9d8188740e00 x1862023066610176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768021.173278:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066610176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768021.173286:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768021.173290:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768021.173293:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066610176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768021.173309:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768021.173324:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768021.173332:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768021/real 1775768021] req@ffff9d8188741180 x1862023066610304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768021.173340:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066610304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768021.173345:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768021.173348:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768021.173350:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768021.173355:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768021.173364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768021.173370:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768021/real 1775768021] req@ffff9d8188743100 x1862023066610432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768021.173377:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066610432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768021.173382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768021.173384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768021.173386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768021.173389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768021.173409:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768021.173416:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768021/real 1775768021] req@ffff9d8188742d80 x1862023066610560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768021.173423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066610560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768021.173430:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768021.173432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768021.173434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768021.173437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768021.173447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768021.173453:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768021/real 1775768021] req@ffff9d8188743b80 x1862023066610688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768021.173460:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066610688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768021.173466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768021.173471:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768021.173473:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768021.813089:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768022.196989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768022.837121:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768023.221036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768023.861073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768024.244997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768024.885066:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768025.268998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768025.909046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768026.292968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768026.292974:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768026.292981:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768026.292985:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768026.292993:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4857 00000100:00080000:3.0:1775768026.293001:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768026.293014:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768026.293049:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066610816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768026.293084:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768026.293089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768026.293091:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768026.293096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4857 00000100:00080000:3.0:1775768026.293099:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768026.293103:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768026.293111:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066610816:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768026.293112:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768026.293126:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066610944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768026.293135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768026.293137:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768026.293139:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768026.293143:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4857 00000100:00080000:3.0:1775768026.293161:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768026.293166:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768026.293168:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775768026.293179:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768026.293196:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066611072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768026.293207:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768026.293208:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768026/real 1775768026] req@ffff9d8188743480 x1862023066610816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768026.293211:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768026.293215:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768026.293224:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4857 00000100:00080000:3.0:1775768026.293229:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768026.293232:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768026.293236:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066610816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768026.293245:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775768026.293246:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775768026.293252:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768026.293255:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066610816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768026.293261:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066611200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768026.293269:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768026.293271:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768026.293273:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768026.293276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610944:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775768026.293280:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4857 00000100:00080000:3.0:1775768026.293284:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768026.293290:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768026.293294:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775768026.293298:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768026.293306:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066611328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768026.293309:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768026/real 1775768026] req@ffff9d8188740a80 x1862023066610944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768026.293347:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066610944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768026.293353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768026.293356:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768026.293358:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066610944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768026.293363:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768026.293375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768026.293382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768026/real 1775768026] req@ffff9d8188741500 x1862023066611072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768026.293389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066611072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768026.293395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768026.293397:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768026.293399:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768026.293402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768026.293412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768026.293417:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768026/real 1775768026] req@ffff9d8188743b80 x1862023066611200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768026.293424:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066611200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768026.293431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768026.293432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768026.293434:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768026.293443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768026.293452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768026.293458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768026/real 1775768026] req@ffff9d8188742d80 x1862023066611328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768026.293465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066611328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768026.293470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768026.293472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768026.293474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768026.933109:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768027.316999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768027.957086:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768028.340986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768028.981131:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768029.364953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768030.005074:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768030.388971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768031.029018:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768031.412936:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768031.412944:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1775768031.412946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768031.412948:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768031.412957:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4862 00000100:00080000:3.0:1775768031.412966:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768031.412972:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768031.413004:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066611456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768031.413026:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768031.413029:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768031.413032:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768031.413036:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4862 00000100:00080000:3.0:1775768031.413040:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768031.413044:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768031.413053:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066611456:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768031.413054:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768031.413108:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066611584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768031.413140:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768031.413145:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768031.413147:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768031.413151:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4862 00000100:00080000:3.0:1775768031.413155:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768031.413160:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768031.413162:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768031.413169:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768031.413182:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066611712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768031.413189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768031.413193:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768031.413194:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768031.413213:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4862 00000100:00080000:3.0:1775768031.413216:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768031.413218:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768031.413223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768031.413231:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768031/real 1775768031] req@ffff9d8188743100 x1862023066611456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768031.413235:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066611840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768031.413247:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768031.413252:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768031.413256:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768031.413262:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4862 00000100:00080000:3.0:1775768031.413266:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768031.413269:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066611456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768031.413274:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768031.413284:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775768031.413285:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768031.413291:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768031.413295:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066611456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768031.413301:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066611968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768031.413315:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768031.413334:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768031.413343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768031/real 1775768031] req@ffff9d8188741180 x1862023066611584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768031.413351:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066611584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768031.413356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768031.413359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768031.413361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768031.413364:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768031.413374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768031.413379:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768031/real 1775768031] req@ffff9d8188740e00 x1862023066611712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768031.413391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066611712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768031.413396:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768031.413398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768031.413399:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768031.413403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768031.413413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768031.413418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768031/real 1775768031] req@ffff9d8188742300 x1862023066611840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768031.413425:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066611840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768031.413432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768031.413434:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768031.413435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768031.413439:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768031.413447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768031.413452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768031/real 1775768031] req@ffff9d8188742680 x1862023066611968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768031.413459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066611968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768031.413469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768031.413471:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768031.413472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066611968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768032.053021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768032.436986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768033.077056:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768033.460990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768034.101022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768034.485053:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768035.125071:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768035.509024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768036.149017:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768036.532996:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768036.532997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768036.533006:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768036.533011:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768036.533021:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4868 00000100:00080000:1.0:1775768036.533032:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768036.533040:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768036.533075:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066612096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768036.533097:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768036.533100:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768036.533101:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768036.533104:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4868 00000100:00080000:1.0:1775768036.533107:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768036.533110:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768036.533119:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768036.533127:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066612096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768036.533131:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066612224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768036.533179:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768036.533195:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768036.533199:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768036.533201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768036.533206:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4868 00000100:00080000:1.0:1775768036.533210:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768036.533214:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768036.533219:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768036/real 1775768036] req@ffff9d8190550e00 x1862023066612096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768036.533219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768036.533235:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066612352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768036.533239:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066612096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768036.533253:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768036.533260:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768036.533262:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768036.533264:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768036.533267:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4868 00000100:00080000:1.0:1775768036.533272:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768036.533275:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:1.0:1775768036.533277:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768036.533283:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768036.533286:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066612096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768036.533293:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066612480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768036.533300:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768036.533302:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768036.533304:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768036.533306:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4868 00000100:00080000:1.0:1775768036.533309:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768036.533312:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768036.533315:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768036.533317:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768036.533324:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066612608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768036.533341:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768036.533354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768036/real 1775768036] req@ffff9d8190551880 x1862023066612224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768036.533367:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066612224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768036.533373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768036.533376:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768036.533378:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768036.533381:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768036.533392:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768036.533398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768036/real 1775768036] req@ffff9d8190551500 x1862023066612352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768036.533405:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066612352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768036.533410:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768036.533412:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768036.533414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768036.533417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768036.533427:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768036.533432:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768036/real 1775768036] req@ffff9d8190553100 x1862023066612480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768036.533448:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066612480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768036.533459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768036.533461:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768036.533463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768036.533472:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768036.533483:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768036.533489:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768036/real 1775768036] req@ffff9d8190553480 x1862023066612608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768036.533496:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066612608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768036.533502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768036.533504:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768036.533506:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768037.173049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768037.557005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768038.197023:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768038.580994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768039.221098:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768039.604974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768040.245045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768040.628957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768041.269064:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768041.652961:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768041.652964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768041.652972:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768041.652975:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768041.652983:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4873 00000100:00080000:1.0:1775768041.652993:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768041.652998:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768041.653030:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066612736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768041.653052:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768041.653055:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768041.653056:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768041.653059:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4873 00000100:00080000:1.0:1775768041.653061:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768041.653064:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768041.653071:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768041.653080:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066612864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768041.653090:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768041.653097:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066612736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768041.653098:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768041.653102:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768041.653109:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4873 00000100:00080000:1.0:1775768041.653113:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768041.653118:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768041.653124:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768041.653138:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066612992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768041.653149:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768041.653154:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775768041.653155:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768041.653157:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768041.653164:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4873 00000100:00080000:1.0:1775768041.653169:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768041.653172:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768041.653177:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768041.653188:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066613120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768041.653196:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768041.653199:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768041/real 1775768041] req@ffff9d8190553480 x1862023066612736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768041.653203:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768041.653207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768041.653217:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4873 00000100:00080000:1.0:1775768041.653226:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768041.653232:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066612736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768041.653233:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768041.653243:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775768041.653245:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768041.653251:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768041.653254:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066612736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768041.653259:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066613248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768041.653276:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768041.653292:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768041.653301:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768041/real 1775768041] req@ffff9d8190553100 x1862023066612864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768041.653309:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066612864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768041.653315:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768041.653317:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768041.653319:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768041.653322:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768041.653333:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768041.653338:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768041/real 1775768041] req@ffff9d8190551500 x1862023066612992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768041.653345:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066612992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768041.653355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768041.653357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768041.653359:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066612992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768041.653362:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768041.653372:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768041.653377:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768041/real 1775768041] req@ffff9d8190551880 x1862023066613120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768041.653384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066613120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768041.653401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768041.653404:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768041.653408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768041.653414:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768041.653435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768041.653446:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768041/real 1775768041] req@ffff9d8190550e00 x1862023066613248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768041.653457:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066613248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768041.653463:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768041.653465:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768041.653468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768042.293043:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768042.676988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768043.317038:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768043.700962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768044.341061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768044.724937:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768045.365002:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768045.749000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768046.389093:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768046.773009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768046.773014:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768046.773024:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768046.773029:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768046.773038:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4878 00000100:00080000:2.0:1775768046.773046:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768046.773052:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768046.773083:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066613376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768046.773104:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768046.773107:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768046.773108:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768046.773110:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4878 00000100:00080000:2.0:1775768046.773113:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768046.773115:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768046.773122:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768046.773130:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066613504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768046.773138:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768046.773140:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768046.773141:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768046.773145:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4878 00000100:00080000:2.0:1775768046.773149:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768046.773151:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066613376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768046.773156:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768046.773164:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768046.773183:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066613632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768046.773193:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768046.773197:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768046.773199:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768046.773204:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4878 00000100:00080000:2.0:1775768046.773207:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768046.773212:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768046.773217:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768046.773228:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066613760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768046.773243:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768046.773250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1775768046.773253:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775768046.773254:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768046.773263:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4878 00000100:00080000:2.0:1775768046.773267:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768046.773270:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768046.773273:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775768046.773284:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066613888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768046.773327:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768046/real 1775768046] req@ffff9d81984a0700 x1862023066613376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768046.773346:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066613376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768046.773355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768046.773359:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768046.773362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066613376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768046.773385:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768046.773413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768046.773422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768046/real 1775768046] req@ffff9d81984a0000 x1862023066613504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768046.773431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066613504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768046.773440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768046.773444:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768046.773448:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768046.773454:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768046.773475:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768046.773488:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768046/real 1775768046] req@ffff9d81984a2680 x1862023066613632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768046.773500:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066613632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768046.773509:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768046.773513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768046.773516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768046.773523:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768046.773543:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768046.773554:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768046/real 1775768046] req@ffff9d81984a0e00 x1862023066613760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768046.773563:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066613760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768046.773570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768046.773573:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768046.773576:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768046.773583:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768046.773604:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768046.773615:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768046/real 1775768046] req@ffff9d81984a1f80 x1862023066613888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768046.773629:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066613888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768046.773639:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768046.773642:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768046.773646:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066613888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768047.413048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768047.796990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768048.437049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768048.820987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768049.461036:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768049.845006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768050.485090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768050.868948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768051.509050:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768051.892976:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768051.892979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768051.892985:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768051.892989:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768051.892995:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4883 00000100:00080000:3.0:1775768051.893004:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768051.893009:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768051.893038:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066614016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768051.893057:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768051.893059:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768051.893061:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768051.893063:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4883 00000100:00080000:3.0:1775768051.893065:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768051.893067:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768051.893074:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768051.893086:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066614144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768051.893091:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066614016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768051.893103:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768051.893106:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768051.893108:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768051.893113:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4883 00000100:00080000:3.0:1775768051.893117:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768051.893121:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768051.893126:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768051.893138:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066614272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768051.893153:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768051.893154:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768051.893158:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768051.893160:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768051.893166:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4883 00000100:00080000:3.0:1775768051.893171:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768051.893175:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768051.893181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768051.893191:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768051/real 1775768051] req@ffff9d8188741880 x1862023066614016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768051.893196:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066614400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768051.893207:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066614016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768051.893208:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768051.893217:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768051.893221:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768051.893223:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768051.893228:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4883 00000100:00080000:0.0:1775768051.893229:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768051.893235:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066614016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768051.893235:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768051.893243:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768051.893249:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768051.893257:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775768051.893263:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066614528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768051.893275:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768051.893285:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768051/real 1775768051] req@ffff9d8188743800 x1862023066614144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768051.893298:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066614144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768051.893305:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768051.893308:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768051.893311:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768051.893317:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768051.893335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768051.893345:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768051/real 1775768051] req@ffff9d8188741f80 x1862023066614272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768051.893353:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066614272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768051.893358:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768051.893360:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768051.893362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768051.893365:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768051.893377:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768051.893383:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768051/real 1775768051] req@ffff9d8188740000 x1862023066614400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768051.893389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066614400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768051.893395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768051.893397:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768051.893398:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768051.893407:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768051.893416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768051.893421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768051/real 1775768051] req@ffff9d8188740700 x1862023066614528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768051.893427:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066614528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768051.893432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768051.893433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768051.893435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768052.533061:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768052.916964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768053.557017:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768053.940988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768054.581027:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768054.964987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768055.605048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768055.988982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768056.629062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768057.012993:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768057.012995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768057.013004:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768057.013009:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768057.013019:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4888 00000100:00080000:3.0:1775768057.013030:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768057.013036:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768057.013083:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066614656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768057.013112:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768057.013114:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768057.013116:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768057.013119:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4888 00000100:00080000:3.0:1775768057.013122:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768057.013125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768057.013135:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768057.013143:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066614656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768057.013147:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066614784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768057.013155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768057.013157:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768057.013158:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768057.013162:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4888 00000100:00080000:3.0:1775768057.013164:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768057.013166:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768057.013168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768057.013176:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066614912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768057.013213:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768057.013216:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768057.013218:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768057.013221:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4888 00000100:00080000:3.0:1775768057.013224:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768057.013226:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768057.013230:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768057.013240:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066615040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768057.013247:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768057.013249:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768057.013251:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768057.013255:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4888 00000100:00080000:3.0:1775768057.013258:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768057.013261:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614784:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775768057.013262:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768057.013266:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768057.013274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066615168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768057.013309:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768057.013327:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768057.013351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768057.653048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768058.036970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768058.677041:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768059.061060:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768059.701066:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768060.084966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775768060.085104:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775768060.085142:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775768060.085158:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768060.085195:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768060.085214:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768060.085222:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768060.085230:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775768060.085298:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768057/real 1775768060] req@ffff9d8188740000 x1862023066615168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768060.085319:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066615168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768060.085331:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768060.085335:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768060.085339:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615168:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768060.085345:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768057/real 1775768060] req@ffff9d8188741c00 x1862023066614784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768060.085357:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066614784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768060.085363:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768060.085366:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768060.085370:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768060.085375:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768057/real 1775768060] req@ffff9d8188740380 x1862023066614912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768060.085385:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066614912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768060.085391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768060.085394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768060.085397:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066614912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768060.085403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768057/real 1775768060] req@ffff9d8188740700 x1862023066615040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768060.085413:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066615040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768060.085421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768060.085424:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768060.085427:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768060.085432:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768057/real 1775768060] req@ffff9d8188742a00 x1862023066614656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768060.085442:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066614656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768060.085448:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768060.085451:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768060.085454:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066614656:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775768060.725018:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768061.108968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768061.749058:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768062.132984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768062.132992:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768062.132997:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768062.133001:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768062.133007:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4893 00000100:00080000:3.0:1775768062.133016:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768062.133022:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768062.133053:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066615296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768062.133077:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768062.133081:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768062.133083:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768062.133087:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4893 00000100:00080000:3.0:1775768062.133091:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768062.133095:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768062.133104:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768062.133114:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066615296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768062.133119:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066615424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768062.133130:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768062.133134:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768062.133135:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768062.133139:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4893 00000100:00080000:3.0:1775768062.133143:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768062.133148:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768062.133153:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768062.133170:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066615552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768062.133177:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768062.133185:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768062.133189:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768062.133191:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768062.133196:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4893 00000100:00080000:3.0:1775768062.133201:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768062.133205:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768062.133209:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768062.133222:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768062/real 1775768062] req@ffff9d8188741f80 x1862023066615296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768062.133224:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066615680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768062.133250:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768062.133253:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768062.133255:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768062.133259:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066615296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768062.133262:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4893 00000100:00080000:3.0:1775768062.133269:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768062.133275:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775768062.133276:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1775768062.133281:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775768062.133283:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768062.133288:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066615296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768062.133295:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066615808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768062.133312:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768062.133335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768062.133346:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768062/real 1775768062] req@ffff9d8188743800 x1862023066615424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768062.133358:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066615424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768062.133365:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768062.133368:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768062.133371:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768062.133376:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768062.133394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768062.133403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768062/real 1775768062] req@ffff9d8188741880 x1862023066615552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768062.133415:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066615552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768062.133421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768062.133424:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768062.133426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768062.133431:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768062.133447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768062.133456:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768062/real 1775768062] req@ffff9d8188742680 x1862023066615680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768062.133465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066615680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768062.133473:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768062.133476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768062.133479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768062.133483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768062.133498:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768062.133505:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768062/real 1775768062] req@ffff9d8188742300 x1862023066615808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768062.133527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066615808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768062.133534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768062.133537:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768062.133540:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066615808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768062.773024:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768063.156944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768063.797028:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768064.180993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768064.820990:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768065.204930:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768065.845039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768066.228958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768066.869004:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768067.253032:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768067.253038:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768067.253045:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768067.253051:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768067.253077:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4898 00000100:00080000:1.0:1775768067.253088:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768067.253096:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768067.253133:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066615936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768067.253156:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768067.253158:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768067.253159:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768067.253162:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4898 00000100:00080000:1.0:1775768067.253165:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768067.253168:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768067.253176:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768067.253185:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066616064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768067.253193:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768067.253194:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066615936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768067.253196:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768067.253199:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768067.253205:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4898 00000100:00080000:1.0:1775768067.253208:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768067.253210:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768067.253213:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768067.253222:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066616192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768067.253228:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768067.253231:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768067.253232:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768067.253234:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4898 00000100:00080000:1.0:1775768067.253243:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768067.253246:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768067.253249:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768067.253253:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768067.253259:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066616320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768067.253267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768067.253269:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768067.253271:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768067.253273:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4898 00000100:00080000:1.0:1775768067.253275:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768067.253277:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768067.253280:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768067.253289:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066616448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768067.253295:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768067/real 1775768067] req@ffff9d8190550e00 x1862023066615936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768067.253331:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066615936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768067.253340:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768067.253343:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768067.253346:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066615936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768067.253364:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768067.253380:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768067.253387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768067/real 1775768067] req@ffff9d8190551880 x1862023066616064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768067.253397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066616064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768067.253406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768067.253410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768067.253413:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768067.253419:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768067.253438:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768067.253449:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768067/real 1775768067] req@ffff9d8190551500 x1862023066616192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768067.253461:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066616192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768067.253467:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768067.253470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768067.253472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768067.253476:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768067.253490:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768067.253497:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768067/real 1775768067] req@ffff9d8190553100 x1862023066616320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768067.253508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066616320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768067.253515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768067.253517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768067.253518:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768067.253546:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768067.253561:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768067.253568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768067/real 1775768067] req@ffff9d8190553480 x1862023066616448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768067.253575:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066616448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768067.253581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768067.253583:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768067.253585:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768067.893028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768068.276958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768068.917068:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768069.300979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768069.941061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768070.324978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768070.965048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768071.348979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768071.989064:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768072.372984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768072.373015:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768072.373022:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768072.373027:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768072.373037:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4903 00000100:00080000:3.0:1775768072.373047:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768072.373054:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768072.373092:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066616576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768072.373121:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768072.373125:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768072.373127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768072.373132:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4903 00000100:00080000:3.0:1775768072.373136:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768072.373140:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768072.373150:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768072.373152:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066616576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768072.373164:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066616704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768072.373177:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768072.373181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768072.373183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768072.373188:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4903 00000100:00080000:3.0:1775768072.373200:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768072.373203:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768072.373208:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775768072.373210:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768072.373224:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066616832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768072.373235:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768072.373239:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768072.373241:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768072.373247:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4903 00000100:00080000:3.0:1775768072.373252:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768072.373255:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768072/real 1775768072] req@ffff9d8188740e00 x1862023066616576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768072.373260:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768072.373271:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768072.373277:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066616576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768072.373287:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775768072.373287:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066616960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768072.373294:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768072.373300:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066616576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768072.373309:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768072.373313:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768072.373315:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768072.373321:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4903 00000100:00100000:0.0:1775768072.373323:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616704:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775768072.373327:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768072.373333:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768072.373339:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775768072.373344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768072.373355:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066617088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768072.373356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768072/real 1775768072] req@ffff9d8188741180 x1862023066616704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768072.373388:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066616704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768072.373398:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768072.373402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768072.373407:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768072.373415:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768072.373435:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768072.373447:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768072/real 1775768072] req@ffff9d8188743100 x1862023066616832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768072.373468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066616832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768072.373514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768072.373518:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768072.373521:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768072.373529:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768072.373552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768072.373563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768072/real 1775768072] req@ffff9d8188742d80 x1862023066616960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768072.373575:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066616960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768072.373586:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768072.373589:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768072.373593:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066616960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768072.373609:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768072.373628:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768072.373638:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768072/real 1775768072] req@ffff9d8188743b80 x1862023066617088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768072.373645:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066617088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768072.373655:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768072.373658:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768072.373660:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768073.013084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768073.396979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768074.037085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768074.420958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768075.061039:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768075.445046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768076.085067:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768076.468962:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768077.108999:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768077.492977:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768077.492985:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768077.492988:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768077.492996:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4909 00000100:00080000:3.0:1775768077.493006:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768077.493011:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768077.493049:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066617216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768077.493086:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768077.493093:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768077.493096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768077.493098:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768077.493102:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4909 00000100:00080000:3.0:1775768077.493105:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768077.493111:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768077.493121:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768077.493137:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066617344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768077.493147:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066617216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768077.493150:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768077.493155:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768077.493157:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768077.493163:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4909 00000100:00080000:3.0:1775768077.493168:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768077.493172:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768077.493179:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768077.493192:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066617472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768077.493200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775768077.493203:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768077.493204:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768077.493207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768077.493211:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4909 00000100:00080000:3.0:1775768077.493215:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768077.493219:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768077.493223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768077.493232:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066617600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768077.493239:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768077.493241:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768077.493243:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768077.493247:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768077/real 1775768077] req@ffff9d8188741500 x1862023066617216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768077.493247:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4909 00000100:00080000:3.0:1775768077.493251:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768077.493256:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768077.493260:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768077.493270:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066617728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768077.493279:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066617216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768077.493292:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768077.493297:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768077.493301:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066617216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768077.493323:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768077.493348:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768077.493362:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768077/real 1775768077] req@ffff9d8188740a80 x1862023066617344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768077.493376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066617344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768077.493386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768077.493390:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768077.493393:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768077.493400:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768077.493442:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768077.493455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768077/real 1775768077] req@ffff9d8188743480 x1862023066617472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768077.493469:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066617472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768077.493478:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768077.493480:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768077.493483:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768077.493487:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768077.493503:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768077.493511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768077/real 1775768077] req@ffff9d8188743b80 x1862023066617600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768077.493518:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066617600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768077.493525:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768077.493527:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768077.493529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768077.493532:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768077.493552:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768077.493558:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768077/real 1775768077] req@ffff9d8188742d80 x1862023066617728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768077.493565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066617728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768077.493572:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768077.493574:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768077.493576:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768078.133067:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768078.516986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768079.157059:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768079.541045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768080.181055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768080.564960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768081.205069:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768081.588980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768082.228995:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768082.612933:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768082.612934:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768082.612942:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768082.612945:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768082.612953:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4914 00000100:00080000:3.0:1775768082.612961:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768082.612967:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768082.612999:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066617856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768082.613021:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768082.613024:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768082.613025:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768082.613028:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4914 00000100:00080000:3.0:1775768082.613030:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768082.613032:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768082.613040:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768082.613047:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066617984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768082.613053:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768082.613056:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768082.613057:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768082.613060:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4914 00000100:00080000:3.0:1775768082.613061:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768082.613063:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768082.613066:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768082.613072:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066618112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768082.613078:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768082.613081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768082.613082:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768082.613085:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4914 00000100:00080000:3.0:1775768082.613087:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768082.613090:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768082.613093:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768082.613100:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066618240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768082.613106:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066617856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768082.613109:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768082.613114:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768082.613116:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768082.613119:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4914 00000100:00080000:3.0:1775768082.613121:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768082.613123:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768082.613126:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768082.613134:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066618368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768082.613167:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768082.613203:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768082/real 1775768082] req@ffff9d8188743100 x1862023066617856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768082.613216:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066617856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768082.613223:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768082.613226:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768082.613229:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066617856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768082.613234:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768082.613248:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768082.613255:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768082/real 1775768082] req@ffff9d8188741180 x1862023066617984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768082.613262:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066617984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768082.613267:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768082.613269:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768082.613271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066617984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768082.613275:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768082.613285:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768082.613290:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768082/real 1775768082] req@ffff9d8188740e00 x1862023066618112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768082.613298:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066618112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768082.613303:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768082.613304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768082.613306:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768082.613325:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768082.613335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768082.613341:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768082/real 1775768082] req@ffff9d8188742300 x1862023066618240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768082.613348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066618240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768082.613354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768082.613355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768082.613357:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768082.613361:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768082.613370:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768082.613396:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768082/real 1775768082] req@ffff9d8188742680 x1862023066618368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768082.613406:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066618368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768082.613412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768082.613414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768082.613416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768083.253051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768083.637031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768084.277068:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768084.660979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768085.301104:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768085.685032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768086.325066:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768086.709049:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768087.349092:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768087.732982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768087.733002:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768087.733009:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768087.733015:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768087.733027:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4919 00000100:00080000:1.0:1775768087.733040:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768087.733049:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768087.733116:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066618496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768087.733145:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768087.733148:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768087.733149:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768087.733152:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4919 00000100:00080000:1.0:1775768087.733155:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768087.733158:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768087.733168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768087.733178:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066618624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768087.733185:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066618496:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768087.733199:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768087.733201:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768087.733202:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768087.733206:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4919 00000100:00080000:1.0:1775768087.733208:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768087.733211:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768087.733215:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768087.733224:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066618752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768087.733231:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768087.733233:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768087.733235:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768087.733237:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4919 00000100:00080000:1.0:1775768087.733259:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768087.733261:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768087.733264:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768087.733271:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066618880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768087.733278:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768087.733283:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1775768087.733284:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768087.733284:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768087.733291:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4919 00000100:00080000:1.0:1775768087.733294:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768087.733297:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768087.733300:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768087.733306:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066619008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768087.733366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768087/real 1775768087] req@ffff9d8190553480 x1862023066618496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768087.733391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066618496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768087.733400:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768087.733404:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768087.733408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066618496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768087.733431:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768087.733458:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768087.733467:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768087/real 1775768087] req@ffff9d8190553100 x1862023066618624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768087.733476:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066618624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768087.733482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768087.733484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768087.733487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768087.733492:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768087.733511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768087.733531:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768087/real 1775768087] req@ffff9d8190551500 x1862023066618752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768087.733546:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066618752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768087.733561:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768087.733564:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768087.733566:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768087.733570:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768087.733587:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768087.733594:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768087/real 1775768087] req@ffff9d8190551880 x1862023066618880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768087.733602:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066618880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768087.733609:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768087.733611:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768087.733613:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066618880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768087.733616:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768087.733626:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768087.733631:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768087/real 1775768087] req@ffff9d8190550e00 x1862023066619008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768087.733638:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066619008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768087.733644:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768087.733646:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768087.733648:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768088.373036:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768088.757027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768089.397069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768089.780964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768090.421028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768090.804981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768091.445051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768091.828972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768092.469030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768092.852977:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768092.852981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768092.852988:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768092.852992:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768092.852999:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4924 00000100:00080000:1.0:1775768092.853009:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768092.853015:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768092.853049:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066619136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768092.853073:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768092.853076:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768092.853077:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768092.853080:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4924 00000100:00080000:1.0:1775768092.853082:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768092.853084:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768092.853091:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768092.853101:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066619264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768092.853108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768092.853113:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768092.853114:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768092.853117:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4924 00000100:00080000:1.0:1775768092.853119:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768092.853124:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768092.853125:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066619136:192.168.202.105@tcp:250:kworker.0 00000008:00100000:1.0:1775768092.853127:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768092.853138:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066619392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768092.853145:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768092.853147:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768092.853148:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768092.853151:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4924 00000100:00080000:1.0:1775768092.853153:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768092.853155:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768092.853158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768092.853165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066619520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768092.853171:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768092.853173:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768092.853175:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768092.853177:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4924 00000100:00080000:1.0:1775768092.853179:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768092.853181:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768092.853185:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775768092.853190:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768092.853195:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066619648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768092.853242:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768092/real 1775768092] req@ffff9d8190550e00 x1862023066619136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768092.853263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066619136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768092.853272:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768092.853300:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768092.853303:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066619136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768092.853309:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768092.853328:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768092.853337:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768092/real 1775768092] req@ffff9d8190551880 x1862023066619264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768092.853346:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066619264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768092.853351:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768092.853354:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768092.853356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768092.853373:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768092.853383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768092.853389:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768092/real 1775768092] req@ffff9d8190551500 x1862023066619392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768092.853397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066619392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768092.853401:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768092.853403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768092.853405:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768092.853408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768092.853418:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768092.853424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768092/real 1775768092] req@ffff9d8190553100 x1862023066619520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768092.853431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066619520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768092.853438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768092.853439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768092.853441:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768092.853445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768092.853454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768092.853459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768092/real 1775768092] req@ffff9d8190553480 x1862023066619648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768092.853466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066619648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768092.853471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768092.853473:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768092.853475:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768093.493075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768093.876955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768094.517032:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768094.901004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768095.541046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768095.924970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768096.565059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768096.949005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768097.589044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768097.972979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768097.973036:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768097.973045:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768097.973050:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768097.973060:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4929 00000100:00080000:3.0:1775768097.973073:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768097.973080:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768097.973128:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066619776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768097.973164:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768097.973168:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768097.973171:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768097.973175:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4929 00000100:00080000:3.0:1775768097.973180:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768097.973185:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768097.973198:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768097.973212:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066619776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768097.973219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066619904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768097.973252:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768097.973256:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768097.973258:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768097.973262:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4929 00000100:00080000:3.0:1775768097.973266:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768097.973270:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768097.973277:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768097.973292:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066620032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768097.973298:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768097.973306:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775768097.973307:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768097.973308:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768097.973322:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4929 00000100:00080000:3.0:1775768097.973328:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768097.973332:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768097.973336:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768097.973349:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066620160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768097.973356:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768097.973360:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768097.973361:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768097.973365:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4929 00000100:00080000:3.0:1775768097.973368:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768097.973371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768097/real 1775768097] req@ffff9d8188741880 x1862023066619776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768097.973373:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768097.973379:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768097.973391:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066620288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768097.973415:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066619776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768097.973425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768097.973429:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768097.973433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066619776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768097.973455:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768097.973474:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768097.973482:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768097/real 1775768097] req@ffff9d8188743800 x1862023066619904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768097.973490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066619904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768097.973496:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768097.973498:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768097.973500:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066619904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768097.973505:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768097.973515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768097.973520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768097/real 1775768097] req@ffff9d8188741f80 x1862023066620032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768097.973527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066620032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768097.973532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768097.973534:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768097.973536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768097.973540:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768097.973557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768097.973567:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768097/real 1775768097] req@ffff9d8188742a00 x1862023066620160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768097.973579:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066620160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768097.973590:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768097.973594:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768097.973598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768097.973604:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768097.973622:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768097.973629:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768097/real 1775768097] req@ffff9d8188740700 x1862023066620288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768097.973637:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066620288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768097.973643:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768097.973645:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768097.973647:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768098.613045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768098.996988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768099.637022:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768100.021002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768100.661041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768101.044980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768101.685024:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768102.068984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768102.709058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768103.092977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768103.093006:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768103.093013:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768103.093018:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768103.093028:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4934 00000100:00080000:1.0:1775768103.093039:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768103.093047:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768103.093102:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066620416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768103.093131:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768103.093135:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768103.093137:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768103.093142:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4934 00000100:00080000:1.0:1775768103.093147:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768103.093151:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768103.093161:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768103.093165:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066620416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768103.093189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066620544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768103.093202:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768103.093206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768103.093208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768103.093212:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4934 00000100:00080000:1.0:1775768103.093225:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768103.093231:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768103.093232:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768103.093240:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768103.093255:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066620672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768103.093267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768103.093270:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768103/real 1775768103] req@ffff9d8190553480 x1862023066620416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768103.093273:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768103.093277:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768103.093286:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4934 00000100:00100000:0.0:1775768103.093291:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066620416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768103.093291:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768103.093300:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775768103.093305:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775768103.093308:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775768103.093311:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768103.093316:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066620416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768103.093326:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066620800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768103.093335:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620544:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775768103.093342:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768103.093346:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768103.093349:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768103.093355:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4934 00000800:00000100:0.0:1775768103.093357:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768103.093363:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768103.093370:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768103.093372:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768103/real 1775768103] req@ffff9d8190553100 x1862023066620544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768103.093378:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768103.093384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066620544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768103.093391:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1775768103.093395:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066620928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768103.093396:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768103.093402:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768103.093416:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768103.093447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768103.093459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768103/real 1775768103] req@ffff9d8190551500 x1862023066620672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768103.093477:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066620672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768103.093483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768103.093485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768103.093487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768103.093497:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768103.093512:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768103.093518:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768103/real 1775768103] req@ffff9d8190551880 x1862023066620800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768103.093526:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066620800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768103.093532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768103.093534:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768103.093536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768103.093539:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768103.093548:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768103.093553:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768103/real 1775768103] req@ffff9d8190550e00 x1862023066620928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768103.093560:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066620928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768103.093570:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768103.093572:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768103.093574:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066620928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768103.733068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768104.116951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768104.757040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768105.140973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768105.781084:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768106.164995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768106.805076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768107.188972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768107.829016:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768108.213009:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768108.213011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768108.213019:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768108.213024:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768108.213031:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4939 00000100:00080000:3.0:1775768108.213040:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768108.213046:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768108.213079:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066621056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768108.213099:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768108.213101:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768108.213103:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768108.213106:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4939 00000100:00080000:3.0:1775768108.213108:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768108.213110:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768108.213118:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768108.213154:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066621056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768108.213156:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066621184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768108.213170:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768108.213172:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768108.213173:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768108.213176:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4939 00000100:00080000:3.0:1775768108.213179:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768108.213182:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768108.213185:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768108.213195:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066621312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768108.213201:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768108.213204:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768108.213205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768108.213207:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4939 00000100:00080000:3.0:1775768108.213215:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768108.213217:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768108.213220:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768108.213227:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768108.213236:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066621440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768108.213244:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768108.213246:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768108.213247:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768108.213250:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4939 00000100:00080000:3.0:1775768108.213253:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768108.213255:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768108.213260:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768108.213270:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066621568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768108.213275:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768108/real 1775768108] req@ffff9d8188740380 x1862023066621056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768108.213299:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066621056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768108.213311:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768108.213315:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768108.213320:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066621056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768108.213341:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768108.213359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768108.213368:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768108/real 1775768108] req@ffff9d8188741c00 x1862023066621184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768108.213376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066621184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768108.213382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768108.213384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768108.213387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768108.213390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768108.213401:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768108.213407:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768108/real 1775768108] req@ffff9d8188740000 x1862023066621312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768108.213414:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066621312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768108.213419:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768108.213421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768108.213422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768108.213426:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768108.213436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768108.213442:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768108/real 1775768108] req@ffff9d8188740700 x1862023066621440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768108.213449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066621440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768108.213459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768108.213462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768108.213466:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768108.213471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768108.213491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768108.213501:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768108/real 1775768108] req@ffff9d8188742a00 x1862023066621568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768108.213513:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066621568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768108.213520:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768108.213522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768108.213525:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768108.853068:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768109.236963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768109.877066:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768110.261000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768110.901057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768111.284996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768111.925076:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768112.308984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768112.949068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768113.332976:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768113.332982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768113.332989:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768113.332994:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768113.333003:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4944 00000100:00080000:3.0:1775768113.333015:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768113.333022:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768113.333066:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066621696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768113.333108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768113.333111:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768113.333113:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768113.333116:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4944 00000100:00080000:3.0:1775768113.333118:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768113.333121:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768113.333129:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768113.333140:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066621824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768113.333149:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768113.333153:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066621696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768113.333153:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768113.333157:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768113.333165:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4944 00000100:00080000:3.0:1775768113.333167:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768113.333169:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768113.333172:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768113.333181:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066621952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768113.333188:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768113.333191:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768113.333192:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768113.333194:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4944 00000100:00080000:3.0:1775768113.333203:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768113.333206:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768113.333208:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768113.333215:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768113.333219:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066622080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768113.333226:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768113.333228:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768113.333230:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768113.333234:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4944 00000100:00080000:3.0:1775768113.333237:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768113.333240:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768113.333243:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768113.333250:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066622208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768113.333273:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768113/real 1775768113] req@ffff9d8188741f80 x1862023066621696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768113.333289:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066621696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768113.333296:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768113.333300:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768113.333303:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066621696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768113.333321:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768113.333338:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768113.333346:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768113/real 1775768113] req@ffff9d8188743800 x1862023066621824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768113.333354:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066621824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768113.333360:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768113.333362:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768113.333364:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768113.333368:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768113.333378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768113.333383:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768113/real 1775768113] req@ffff9d8188741880 x1862023066621952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768113.333390:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066621952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768113.333395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768113.333397:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768113.333399:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066621952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768113.333403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768113.333413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768113.333418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768113/real 1775768113] req@ffff9d8188742680 x1862023066622080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768113.333430:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066622080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768113.333436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768113.333438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768113.333440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768113.333443:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768113.333452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768113.333458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768113/real 1775768113] req@ffff9d8188742300 x1862023066622208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768113.333465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066622208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768113.333470:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768113.333472:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768113.333473:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768113.972999:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768114.356979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768114.997113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768115.380979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768116.021075:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768116.404991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768117.045076:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768117.428979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768118.069062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768118.452969:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768118.452973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768118.452978:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768118.452982:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768118.452989:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4949 00000100:00080000:3.0:1775768118.452999:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768118.453005:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768118.453043:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066622336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768118.453063:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768118.453065:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768118.453070:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768118.453073:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4949 00000100:00080000:3.0:1775768118.453077:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768118.453080:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768118.453087:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768118.453096:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066622464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768118.453103:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768118.453105:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768118.453106:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768118.453109:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4949 00000100:00080000:3.0:1775768118.453111:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768118.453113:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768118.453116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066622336:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768118.453117:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768118.453126:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066622592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768118.453136:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768118.453138:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768118.453139:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768118.453142:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4949 00000100:00080000:3.0:1775768118.453144:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768118.453146:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768118.453149:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768118.453157:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066622720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768118.453162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768118.453165:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768118.453165:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768118.453168:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4949 00000100:00080000:3.0:1775768118.453170:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768118.453174:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768118.453174:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768118.453179:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768118.453186:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066622848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768118.453212:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768118/real 1775768118] req@ffff9d8188740e00 x1862023066622336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768118.453225:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066622336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768118.453232:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768118.453235:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768118.453237:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066622336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768118.453254:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768118.453267:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768118.453274:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768118/real 1775768118] req@ffff9d8188741180 x1862023066622464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768118.453281:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066622464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768118.453285:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768118.453287:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768118.453289:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768118.453292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768118.453301:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768118.453306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768118/real 1775768118] req@ffff9d8188743100 x1862023066622592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768118.453312:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066622592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768118.453317:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768118.453318:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768118.453320:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768118.453323:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768118.453333:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768118.453337:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768118/real 1775768118] req@ffff9d8188742d80 x1862023066622720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768118.453343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066622720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768118.453349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768118.453350:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768118.453352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768118.453354:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768118.453368:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768118.453376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768118/real 1775768118] req@ffff9d8188743b80 x1862023066622848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768118.453387:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066622848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768118.453395:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768118.453398:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768118.453400:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066622848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768119.093068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768119.476973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768120.117088:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768120.500970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768121.141027:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768121.524966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768122.165062:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768122.548967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768123.189032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768123.573020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768123.573060:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768123.573066:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768123.573069:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768123.573077:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4955 00000100:00080000:1.0:1775768123.573087:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768123.573093:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768123.573128:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066622976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768123.573153:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768123.573159:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768123.573161:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768123.573165:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4955 00000100:00080000:1.0:1775768123.573168:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768123.573172:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768123.573181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768123.573194:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066622976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768123.573195:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066623104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768123.573210:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768123.573213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768123.573215:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768123.573220:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4955 00000100:00080000:1.0:1775768123.573223:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768123.573227:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768123.573232:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768123.573242:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066623232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768123.573251:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768123.573255:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768123.573256:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768123.573260:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4955 00000100:00080000:1.0:1775768123.573263:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768123.573267:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768123.573271:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768123.573285:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066623360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768123.573295:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768123.573299:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768123.573302:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768123.573306:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4955 00000100:00100000:0.0:1775768123.573308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623104:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775768123.573310:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768123.573315:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768123.573320:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768123.573333:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066623488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768123.573336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768123.573357:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768123.573383:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768124.213048:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768124.597005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768125.237080:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768125.621020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768126.261049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768126.644943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775768126.645099:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775768126.645126:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775768126.645138:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775768126.645174:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775768126.645181:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775768126.645188:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775768126.645199:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775768126.645222:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768123/real 1775768126] req@ffff9d8190553480 x1862023066623488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768126.645247:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066623488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768126.645259:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768126.645263:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768126.645268:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623488:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768126.645275:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768123/real 1775768126] req@ffff9d8190551880 x1862023066623104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768126.645288:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066623104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768126.645296:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768126.645299:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768126.645302:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768126.645309:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768123/real 1775768126] req@ffff9d8190551500 x1862023066623232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768126.645320:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066623232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768126.645327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768126.645329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768126.645332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768126.645337:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768123/real 1775768126] req@ffff9d8190553100 x1862023066623360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768126.645347:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066623360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768126.645354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768126.645357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768126.645359:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768126.645365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768123/real 1775768126] req@ffff9d8190550e00 x1862023066622976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768126.645374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066622976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768126.645380:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768126.645382:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768126.645385:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066622976:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775768127.285073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768127.668969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768128.309081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768128.692957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768128.693009:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768128.693015:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768128.693019:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768128.693028:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4960 00000100:00080000:3.0:1775768128.693038:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768128.693044:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768128.693086:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066623616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768128.693109:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768128.693112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768128.693114:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768128.693118:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4960 00000100:00080000:3.0:1775768128.693121:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768128.693125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768128.693134:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768128.693146:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066623744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768128.693156:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066623616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768128.693158:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768128.693165:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768128.693167:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768128.693172:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4960 00000100:00080000:3.0:1775768128.693174:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768128.693178:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768128.693183:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768128.693195:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066623872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768128.693203:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768128.693206:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768128.693208:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768128.693212:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4960 00000100:00080000:3.0:1775768128.693215:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768128.693221:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768128.693221:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768128.693228:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768128.693241:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066624000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768128.693249:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768128.693252:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768128.693254:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768128.693258:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4960 00000100:00080000:3.0:1775768128.693262:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768128.693265:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768128.693269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768128/real 1775768128] req@ffff9d8188743480 x1862023066623616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768128.693272:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768128.693287:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066624128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768128.693299:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066623616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768128.693310:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768128.693315:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768128.693319:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066623616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768128.693340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768128.693360:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768128.693371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768128/real 1775768128] req@ffff9d8188740a80 x1862023066623744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768128.693382:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066623744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768128.693390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768128.693392:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768128.693396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768128.693401:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768128.693417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768128.693426:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768128/real 1775768128] req@ffff9d8188741500 x1862023066623872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768128.693443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066623872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768128.693450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768128.693453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768128.693456:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066623872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768128.693460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768128.693476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768128.693485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768128/real 1775768128] req@ffff9d8188743b80 x1862023066624000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768128.693495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066624000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768128.693504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768128.693508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768128.693510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768128.693515:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768128.693530:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768128.693537:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768128/real 1775768128] req@ffff9d8188742d80 x1862023066624128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768128.693546:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066624128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768128.693564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768128.693567:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768128.693570:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768129.333049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768129.716969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768130.357044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768130.740967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768131.381042:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768131.764980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768132.405116:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768132.788992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768133.429007:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768133.813008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768133.813008:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768133.813019:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768133.813025:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768133.813036:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4965 00000100:00080000:3.0:1775768133.813059:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768133.813068:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768133.813116:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066624256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768133.813154:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768133.813157:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768133.813159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768133.813163:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4965 00000100:00080000:3.0:1775768133.813167:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768133.813171:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768133.813182:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768133.813196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066624384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768133.813201:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066624256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768133.813218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768133.813222:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768133.813223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768133.813228:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4965 00000100:00080000:3.0:1775768133.813232:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768133.813236:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768133.813241:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768133.813253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066624512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768133.813263:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768133.813266:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775768133.813270:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768133.813270:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768133.813277:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4965 00000100:00080000:3.0:1775768133.813282:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768133.813287:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768133.813292:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768133.813304:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066624640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768133.813314:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768133.813318:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775768133.813320:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768133/real 1775768133] req@ffff9d8188743100 x1862023066624256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768133.813321:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768133.813331:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4965 00000100:00080000:3.0:1775768133.813336:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768133.813341:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768133.813343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066624256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768133.813348:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775768133.813354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768133.813358:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768133.813363:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066624256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768133.813364:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066624768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768133.813386:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768133.813402:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768133.813409:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768133/real 1775768133] req@ffff9d8188741180 x1862023066624384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768133.813417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066624384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768133.813422:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768133.813424:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768133.813426:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768133.813430:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768133.813439:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768133.813444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768133/real 1775768133] req@ffff9d8188740e00 x1862023066624512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768133.813451:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066624512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768133.813455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768133.813457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768133.813458:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768133.813461:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768133.813471:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768133.813476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768133/real 1775768133] req@ffff9d8188742300 x1862023066624640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768133.813482:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066624640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768133.813487:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768133.813489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768133.813490:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768133.813494:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768133.813503:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768133.813508:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768133/real 1775768133] req@ffff9d8188742680 x1862023066624768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768133.813514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066624768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768133.813519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768133.813521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768133.813522:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066624768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768134.453042:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768134.836991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768135.477013:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768135.860995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768136.501256:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768136.884999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768137.525065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768137.908987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768138.549048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768138.932953:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768138.932961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768138.932962:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768138.932966:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768138.932975:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4970 00000100:00080000:3.0:1775768138.932989:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768138.932996:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768138.933030:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066624896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768138.933057:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768138.933060:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768138.933061:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768138.933065:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4970 00000100:00080000:3.0:1775768138.933067:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768138.933071:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768138.933079:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768138.933091:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066625024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768138.933101:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768138.933106:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1775768138.933107:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066624896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768138.933108:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768138.933117:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4970 00000100:00080000:3.0:1775768138.933120:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768138.933124:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768138.933129:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768138.933142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066625152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768138.933151:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768138.933154:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768138.933164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775768138.933171:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768138.933172:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4970 00000100:00080000:3.0:1775768138.933177:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768138.933179:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768138.933184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768138.933194:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066625280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768138.933200:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768138.933202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768138.933202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768138.933204:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4970 00000100:00080000:3.0:1775768138.933206:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768138.933208:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768138.933210:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768138.933216:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066625408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768138.933238:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768138/real 1775768138] req@ffff9d8188741880 x1862023066624896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768138.933257:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066624896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768138.933266:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768138.933270:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768138.933274:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066624896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768138.933294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768138.933316:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768138.933329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768138/real 1775768138] req@ffff9d8188743800 x1862023066625024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768138.933346:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066625024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768138.933354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768138.933357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768138.933360:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768138.933366:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768138.933384:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768138.933394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768138/real 1775768138] req@ffff9d8188741f80 x1862023066625152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768138.933406:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066625152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768138.933414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768138.933417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768138.933420:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768138.933425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768138.933452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768138.933464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768138/real 1775768138] req@ffff9d8188742a00 x1862023066625280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768138.933475:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066625280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768138.933489:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768138.933492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768138.933495:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768138.933500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768138.933519:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768138.933529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768138/real 1775768138] req@ffff9d8188740700 x1862023066625408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768138.933539:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066625408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768138.933547:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768138.933550:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768138.933553:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768139.573044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768139.956976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768140.597038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768140.981916:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768141.621057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768142.004977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768142.644999:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768143.029002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768143.669051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768144.052985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768144.052990:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768144.052997:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768144.053002:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768144.053014:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4975 00000100:00080000:2.0:1775768144.053027:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768144.053035:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768144.053079:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066625536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768144.053107:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768144.053109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768144.053111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768144.053116:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4975 00000100:00080000:2.0:1775768144.053119:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768144.053122:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768144.053131:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768144.053139:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066625664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768144.053146:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768144.053149:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768144.053150:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768144.053155:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4975 00000100:00100000:0.0:1775768144.053157:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066625536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768144.053157:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768144.053160:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768144.053167:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768144.053173:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066625792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768144.053179:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768144.053182:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768144.053183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768144.053185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4975 00000100:00080000:2.0:1775768144.053188:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768144.053191:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768144.053194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768144.053204:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066625920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768144.053212:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768144.053215:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768144.053217:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768144.053221:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4975 00000100:00080000:2.0:1775768144.053225:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768144.053234:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768144.053238:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775768144.053242:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775768144.053254:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066626048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768144.053293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768144/real 1775768144] req@ffff9d81984a1180 x1862023066625536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768144.053314:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066625536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768144.053325:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768144.053329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768144.053333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066625536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768144.053360:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768144.053381:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768144.053392:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768144/real 1775768144] req@ffff9d81984a1880 x1862023066625664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768144.053404:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066625664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768144.053412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768144.053415:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768144.053418:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768144.053424:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768144.053441:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768144.053450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768144/real 1775768144] req@ffff9d81984a3800 x1862023066625792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768144.053470:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066625792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768144.053478:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768144.053481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768144.053484:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768144.053490:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768144.053511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768144.053520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768144/real 1775768144] req@ffff9d81984a3100 x1862023066625920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768144.053532:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066625920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768144.053540:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768144.053543:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768144.053546:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066625920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768144.053552:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768144.053578:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768144.053587:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768144/real 1775768144] req@ffff9d81984a1500 x1862023066626048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768144.053598:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066626048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768144.053606:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768144.053609:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768144.053612:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768144.693057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768145.076980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768145.717077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768146.101030:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768146.741035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768147.125027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768147.765056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768148.148941:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768148.789015:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768149.172977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768149.173000:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768149.173007:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768149.173010:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768149.173016:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4980 00000100:00080000:3.0:1775768149.173025:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768149.173036:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768149.173065:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066626176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768149.173081:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768149.173084:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768149.173085:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768149.173087:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4980 00000100:00080000:3.0:1775768149.173089:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768149.173091:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768149.173100:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768149.173108:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066626176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768149.173114:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066626304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768149.173125:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768149.173129:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768149.173130:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768149.173135:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4980 00000100:00080000:3.0:1775768149.173151:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768149.173156:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768149.173159:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775768149.173162:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768149.173173:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066626432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768149.173179:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768149.173182:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768149.173183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768149.173186:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4980 00000100:00080000:3.0:1775768149.173190:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768149.173192:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768149.173196:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768149.173197:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768149/real 1775768149] req@ffff9d8188740000 x1862023066626176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768149.173206:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066626560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768149.173213:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066626176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768149.173219:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768149.173224:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768149.173224:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768149.173228:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768149.173230:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768149.173238:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066626176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768149.173238:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4980 00000100:00080000:3.0:1775768149.173251:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768149.173255:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768149.173256:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626304:192.168.202.105@tcp:38:kworker.0 00000008:00100000:3.0:1775768149.173262:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768149.173274:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066626688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768149.173278:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768149.173289:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768149/real 1775768149] req@ffff9d8188741c00 x1862023066626304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768149.173297:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066626304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768149.173303:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768149.173305:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768149.173307:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768149.173310:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768149.173320:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768149.173325:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768149/real 1775768149] req@ffff9d8188740380 x1862023066626432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768149.173331:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066626432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768149.173339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768149.173341:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768149.173343:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768149.173346:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768149.173361:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768149.173369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768149/real 1775768149] req@ffff9d8188740700 x1862023066626560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768149.173379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066626560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768149.173389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768149.173391:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768149.173394:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768149.173408:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768149.173426:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768149.173436:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768149/real 1775768149] req@ffff9d8188742a00 x1862023066626688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768149.173447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066626688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768149.173455:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768149.173458:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768149.173461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768149.813082:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768150.196932:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768150.837019:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768151.221011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768151.861030:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768152.244980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768152.884991:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768153.269004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768153.909026:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768154.292993:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768154.293001:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775768154.293003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768154.293005:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768154.293019:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4985 00000100:00080000:3.0:1775768154.293027:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768154.293034:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768154.293064:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066626816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768154.293084:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768154.293086:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768154.293087:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768154.293090:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4985 00000100:00080000:3.0:1775768154.293091:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768154.293093:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768154.293099:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768154.293106:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066626944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768154.293113:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768154.293119:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768154.293120:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768154.293123:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4985 00000100:00080000:3.0:1775768154.293126:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768154.293130:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768154.293131:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066626816:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768154.293133:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768154.293142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066627072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768154.293153:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768154.293155:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768154.293156:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768154.293159:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4985 00000100:00080000:3.0:1775768154.293175:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768154.293177:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768154.293180:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768154.293184:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768154.293191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066627200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768154.293200:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768154.293202:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768154.293203:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768154.293205:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4985 00000100:00080000:3.0:1775768154.293208:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768154.293210:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768154.293213:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768154.293220:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066627328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768154.293229:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768154/real 1775768154] req@ffff9d8188741f80 x1862023066626816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768154.293250:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066626816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768154.293259:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768154.293262:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768154.293265:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066626816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768154.293270:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768154.293286:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768154.293293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768154/real 1775768154] req@ffff9d8188743800 x1862023066626944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768154.293300:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066626944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768154.293306:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768154.293307:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768154.293309:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066626944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768154.293326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768154.293344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768154.293354:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768154/real 1775768154] req@ffff9d8188741880 x1862023066627072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768154.293365:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066627072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768154.293373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768154.293376:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768154.293378:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768154.293383:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768154.293397:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768154.293403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768154/real 1775768154] req@ffff9d8188742680 x1862023066627200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768154.293410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066627200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768154.293416:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768154.293418:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768154.293419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768154.293422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768154.293434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768154.293443:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768154/real 1775768154] req@ffff9d8188742300 x1862023066627328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768154.293459:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066627328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768154.293466:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768154.293469:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768154.293472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768154.933041:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768155.316997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768155.957037:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768156.341025:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768156.981045:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768157.364981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768158.005055:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768158.388970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768159.029060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768159.412943:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768159.412950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768159.412950:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768159.412955:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768159.412962:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4990 00000100:00080000:3.0:1775768159.412970:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768159.412975:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768159.413005:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066627456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768159.413025:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768159.413028:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768159.413029:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768159.413031:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4990 00000100:00080000:3.0:1775768159.413033:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768159.413035:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768159.413041:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768159.413047:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066627584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768159.413053:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768159.413055:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768159.413056:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768159.413058:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4990 00000100:00080000:3.0:1775768159.413060:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768159.413062:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768159.413065:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768159.413071:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066627712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768159.413079:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066627456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768159.413083:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768159.413086:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768159.413087:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768159.413090:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4990 00000100:00080000:3.0:1775768159.413091:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768159.413093:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768159.413096:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768159.413104:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066627840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768159.413109:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768159.413111:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768159.413111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768159.413113:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4990 00000100:00080000:3.0:1775768159.413121:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768159.413124:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768159.413128:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775768159.413129:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768159.413138:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066627968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768159.413165:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768159/real 1775768159] req@ffff9d8188740e00 x1862023066627456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768159.413178:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066627456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768159.413185:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768159.413187:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768159.413190:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066627456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768159.413194:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768159.413206:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768159.413213:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768159/real 1775768159] req@ffff9d8188741180 x1862023066627584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768159.413220:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066627584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768159.413225:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768159.413227:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768159.413229:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768159.413244:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768159.413253:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768159.413258:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768159/real 1775768159] req@ffff9d8188743100 x1862023066627712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768159.413264:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066627712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768159.413268:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768159.413270:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768159.413271:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768159.413275:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768159.413284:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768159.413289:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768159/real 1775768159] req@ffff9d8188742d80 x1862023066627840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768159.413295:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066627840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768159.413300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768159.413301:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768159.413303:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768159.413307:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768159.413315:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768159.413319:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768159/real 1775768159] req@ffff9d8188743b80 x1862023066627968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768159.413325:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066627968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768159.413330:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768159.413331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768159.413333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066627968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768160.053023:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768160.436959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768161.077044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768161.460974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768162.101010:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768162.484957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768163.125049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768163.509002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768164.149057:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768164.533017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768164.533030:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768164.533040:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768164.533045:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768164.533057:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 4996 00000100:00080000:2.0:1775768164.533072:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768164.533079:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768164.533129:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066628096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768164.533158:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768164.533163:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768164.533164:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768164.533167:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4996 00000100:00080000:2.0:1775768164.533170:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768164.533172:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768164.533184:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768164.533195:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066628224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768164.533204:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768164.533208:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066628096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768164.533208:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768164.533211:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768164.533218:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4996 00000100:00080000:2.0:1775768164.533221:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768164.533223:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768164.533235:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768164.533249:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066628352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768164.533256:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768164.533259:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768164.533261:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768164.533264:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4996 00000100:00080000:2.0:1775768164.533273:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768164.533276:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768164.533281:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768164.533285:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775768164.533291:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066628480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768164.533299:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768164.533302:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768164.533303:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768164.533306:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 4996 00000100:00080000:2.0:1775768164.533308:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768164.533310:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768164.533313:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775768164.533322:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066628608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768164.533339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768164/real 1775768164] req@ffff9d81984a0a80 x1862023066628096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768164.533356:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066628096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768164.533364:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768164.533368:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768164.533371:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066628096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768164.533394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768164.533410:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768164.533418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768164/real 1775768164] req@ffff9d81984a1500 x1862023066628224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768164.533426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066628224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768164.533432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768164.533435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768164.533437:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768164.533440:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768164.533450:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768164.533455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768164/real 1775768164] req@ffff9d81984a3100 x1862023066628352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768164.533463:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066628352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768164.533468:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768164.533470:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768164.533472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768164.533475:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768164.533486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768164.533492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768164/real 1775768164] req@ffff9d81984a3800 x1862023066628480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768164.533504:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066628480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768164.533510:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768164.533512:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768164.533514:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768164.533517:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768164.533526:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768164.533531:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768164/real 1775768164] req@ffff9d81984a1880 x1862023066628608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768164.533538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066628608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768164.533544:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768164.533546:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768164.533547:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768165.173087:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768165.557093:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768166.197054:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768166.580995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768167.221075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768167.604955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768168.245024:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768168.629001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768169.269057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768169.652941:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768169.652948:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768169.652949:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768169.652954:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768169.652963:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5001 00000100:00080000:2.0:1775768169.652971:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768169.652976:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768169.653006:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066628736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768169.653025:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768169.653027:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768169.653028:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768169.653030:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5001 00000100:00080000:2.0:1775768169.653033:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768169.653035:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768169.653042:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768169.653048:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066628864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768169.653054:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768169.653057:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768169.653058:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768169.653061:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5001 00000100:00080000:2.0:1775768169.653062:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768169.653065:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768169.653068:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768169.653069:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066628736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768169.653075:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066628992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768169.653081:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768169.653083:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768169.653084:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768169.653087:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5001 00000100:00080000:2.0:1775768169.653088:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768169.653090:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768169.653093:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768169.653100:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066629120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768169.653108:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768169.653111:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768169.653112:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768169.653114:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5001 00000100:00080000:2.0:1775768169.653116:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768169.653117:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768169.653120:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775768169.653124:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775768169.653129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066629248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768169.653174:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768169/real 1775768169] req@ffff9d81984a1180 x1862023066628736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768169.653189:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066628736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768169.653200:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768169.653203:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768169.653206:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066628736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768169.653210:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768169.653225:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768169.653232:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768169/real 1775768169] req@ffff9d81984a1f80 x1862023066628864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768169.653239:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066628864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768169.653243:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768169.653245:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768169.653247:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768169.653261:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768169.653270:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768169.653275:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768169/real 1775768169] req@ffff9d81984a0e00 x1862023066628992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768169.653281:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066628992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768169.653285:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768169.653287:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768169.653289:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066628992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768169.653292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768169.653300:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768169.653305:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768169/real 1775768169] req@ffff9d81984a2680 x1862023066629120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768169.653311:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066629120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768169.653316:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768169.653317:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768169.653319:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768169.653322:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768169.653330:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768169.653334:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768169/real 1775768169] req@ffff9d81984a0000 x1862023066629248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768169.653340:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066629248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768169.653345:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768169.653346:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768169.653348:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768170.292991:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768170.676957:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768171.317026:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768171.701004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768172.341030:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768172.725035:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768173.365019:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768173.749022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768174.389012:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768174.772976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768174.772976:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768174.772985:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768174.772990:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768174.772998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5006 00000100:00080000:2.0:1775768174.773006:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768174.773012:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768174.773044:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066629376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768174.773071:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768174.773074:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768174.773076:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768174.773080:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5006 00000100:00080000:2.0:1775768174.773083:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768174.773087:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768174.773095:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768174.773107:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066629376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768174.773107:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066629504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768174.773120:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768174.773122:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768174.773122:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768174.773125:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5006 00000100:00080000:2.0:1775768174.773128:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768174.773129:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768174.773132:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768174.773140:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066629632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768174.773145:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768174.773147:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768174.773148:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768174.773151:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5006 00000100:00080000:2.0:1775768174.773153:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768174.773154:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768174.773159:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775768174.773164:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768174.773174:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066629760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768174.773179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768174.773181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768174.773183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768174.773186:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5006 00000100:00080000:2.0:1775768174.773189:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768174.773191:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768174.773194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775768174.773200:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066629888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768174.773206:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768174/real 1775768174] req@ffff9d81984a0700 x1862023066629376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768174.773220:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066629376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768174.773227:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768174.773231:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768174.773234:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066629376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768174.773250:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768174.773271:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768174.773279:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768174/real 1775768174] req@ffff9d81984a2a00 x1862023066629504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768174.773286:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066629504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768174.773291:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768174.773293:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768174.773295:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768174.773298:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768174.773307:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768174.773312:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768174/real 1775768174] req@ffff9d81984a3480 x1862023066629632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768174.773319:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066629632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768174.773327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768174.773329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768174.773331:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768174.773334:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768174.773343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768174.773347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768174/real 1775768174] req@ffff9d81984a2d80 x1862023066629760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768174.773363:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066629760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768174.773372:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768174.773374:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768174.773377:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768174.773383:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768174.773400:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768174.773410:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768174/real 1775768174] req@ffff9d81984a3b80 x1862023066629888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768174.773418:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066629888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768174.773423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768174.773425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768174.773427:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066629888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768175.412997:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768175.796949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768176.437043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768176.820973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768177.461009:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768177.844976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768178.485095:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768178.868996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768179.509019:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768179.892966:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768179.892967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768179.892978:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768179.892982:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768179.892993:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5011 00000100:00080000:3.0:1775768179.893004:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768179.893013:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768179.893048:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066630016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768179.893074:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768179.893078:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768179.893080:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768179.893084:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5011 00000100:00080000:3.0:1775768179.893088:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768179.893092:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768179.893102:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768179.893116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066630016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768179.893119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066630144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768179.893133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768179.893136:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768179.893138:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768179.893143:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5011 00000100:00080000:3.0:1775768179.893146:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768179.893150:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768179.893155:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768179.893167:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066630272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768179.893179:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768179.893179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768179.893186:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768179.893188:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768179.893193:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5011 00000100:00080000:3.0:1775768179.893212:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768179.893217:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768179.893224:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768179.893226:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768179/real 1775768179] req@ffff9d8188741500 x1862023066630016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768179.893242:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066630400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768179.893247:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066630016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768179.893268:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768179.893269:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768179.893274:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775768179.893276:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775768179.893280:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066630016:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768179.893281:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768179.893289:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5011 00000100:00080000:3.0:1775768179.893294:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768179.893297:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768179.893303:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768179.893306:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775768179.893320:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066630528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768179.893338:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768179.893355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768179/real 1775768179] req@ffff9d8188740a80 x1862023066630144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768179.893365:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066630144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768179.893370:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768179.893373:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768179.893375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768179.893379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768179.893394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768179.893401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768179/real 1775768179] req@ffff9d8188743480 x1862023066630272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768179.893408:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066630272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768179.893414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768179.893416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768179.893418:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768179.893446:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768179.893462:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768179.893470:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768179/real 1775768179] req@ffff9d8188743b80 x1862023066630400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768179.893478:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066630400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768179.893485:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768179.893487:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768179.893489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768179.893500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768179.893510:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768179.893516:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768179/real 1775768179] req@ffff9d8188742d80 x1862023066630528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768179.893526:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066630528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768179.893535:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768179.893538:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768179.893541:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768180.533090:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768180.916997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768181.557043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768181.940987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768182.581070:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768182.964971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768183.605070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768183.988998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768184.629093:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768185.012976:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768185.012983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768185.012988:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768185.012994:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768185.013004:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5016 00000100:00080000:3.0:1775768185.013014:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768185.013022:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768185.013061:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066630656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768185.013090:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768185.013094:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768185.013096:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768185.013100:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5016 00000100:00080000:3.0:1775768185.013105:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768185.013109:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768185.013120:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768185.013133:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066630656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768185.013137:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066630784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768185.013152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768185.013156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768185.013158:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768185.013162:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5016 00000100:00080000:3.0:1775768185.013166:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768185.013170:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768185.013176:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768185.013189:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066630912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768185.013198:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768185.013201:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768185.013205:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768185.013207:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768185.013213:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5016 00000100:00080000:3.0:1775768185.013218:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768185.013222:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768185.013228:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768185.013241:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066631040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768185.013251:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768185.013254:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768185/real 1775768185] req@ffff9d8188743100 x1862023066630656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768185.013255:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768185.013260:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768185.013270:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5016 00000100:00080000:3.0:1775768185.013275:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768185.013279:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066630656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768185.013283:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768185.013293:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775768185.013295:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768185.013302:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768185.013305:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066630656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768185.013311:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066631168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768185.013326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768185.013347:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768185.013355:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768185/real 1775768185] req@ffff9d8188741180 x1862023066630784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768185.013363:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066630784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768185.013369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768185.013371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768185.013374:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768185.013378:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768185.013407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768185.013415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768185/real 1775768185] req@ffff9d8188740e00 x1862023066630912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768185.013423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066630912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768185.013429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768185.013431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768185.013433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066630912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768185.013437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768185.013450:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768185.013455:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768185/real 1775768185] req@ffff9d8188742300 x1862023066631040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768185.013462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066631040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768185.013469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768185.013471:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768185.013472:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768185.013476:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768185.013485:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768185.013491:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768185/real 1775768185] req@ffff9d8188742680 x1862023066631168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768185.013498:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066631168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768185.013503:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768185.013505:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768185.013507:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768185.653073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768186.036995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768186.677025:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768187.060977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768187.701027:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768188.084974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768188.725090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768189.108939:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768189.749012:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768190.132971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768190.132988:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768190.132994:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768190.132998:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768190.133011:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5021 00000100:00080000:0.0:1775768190.133020:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768190.133026:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768190.133060:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066631296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768190.133085:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768190.133089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768190.133090:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768190.133094:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5021 00000100:00080000:0.0:1775768190.133097:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768190.133102:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768190.133110:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768190.133119:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066631296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768190.133124:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066631424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768190.133135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768190.133138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768190.133140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768190.133144:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5021 00000100:00080000:0.0:1775768190.133147:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768190.133151:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768190.133155:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768190.133164:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066631552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768190.133173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768190.133176:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768190.133178:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768190.133181:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5021 00000100:00080000:0.0:1775768190.133184:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768190.133187:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768190.133191:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768190.133201:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066631680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768190.133209:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768190.133212:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768190.133214:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768190.133219:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5021 00000100:00080000:0.0:1775768190.133222:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768190.133225:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768190.133230:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768190.133242:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768190.133243:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066631808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768190.133281:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768190.133308:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768190.133350:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768190.773028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768191.156988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768191.797022:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768192.180984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768192.821052:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768193.204981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775768193.205098:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775768193.205134:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775768193.205149:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768193.205183:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768193.205191:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768193.205197:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768193.205206:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775768193.205238:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768190/real 1775768193] req@ffff9d81901ad180 x1862023066631808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768193.205262:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066631808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768193.205279:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768193.205284:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768193.205288:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631808:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768193.205295:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768190/real 1775768193] req@ffff9d81901aed80 x1862023066631424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768193.205331:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066631424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768193.205339:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768193.205343:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768193.205345:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768193.205352:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768190/real 1775768193] req@ffff9d81901af100 x1862023066631552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768193.205362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066631552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768193.205374:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768193.205377:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768193.205380:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768193.205386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768190/real 1775768193] req@ffff9d81901ae680 x1862023066631680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768193.205396:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066631680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768193.205404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768193.205407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768193.205410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066631680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768193.205415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768190/real 1775768193] req@ffff9d81901afb80 x1862023066631296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768193.205425:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066631296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768193.205432:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768193.205435:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768193.205437:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066631296:192.168.202.105@tcp:250:kworker.0 00010000:00010000:0.0:1775768193.844977:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768194.228982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768194.869027:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768195.252958:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768195.252959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768195.252967:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768195.252971:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768195.252979:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5026 00000100:00080000:0.0:1775768195.252989:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768195.252994:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768195.253021:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066631936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768195.253042:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768195.253044:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768195.253045:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768195.253048:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5026 00000100:00080000:0.0:1775768195.253050:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768195.253052:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768195.253059:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768195.253067:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066632064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768195.253073:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066631936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768195.253077:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768195.253082:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768195.253083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768195.253088:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5026 00000100:00080000:0.0:1775768195.253090:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768195.253094:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768195.253099:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768195.253121:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066632192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768195.253130:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768195.253134:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768195.253136:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768195.253141:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5026 00000800:00000100:1.0:1775768195.253142:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775768195.253148:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768195.253153:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768195.253159:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768195.253171:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066632320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768195.253180:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768195.253183:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768195.253185:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768195.253189:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5026 00000100:00080000:0.0:1775768195.253193:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768195.253196:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768195/real 1775768195] req@ffff9d81901afb80 x1862023066631936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768195.253200:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768195.253207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768195.253223:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066632448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768195.253224:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066631936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768195.253233:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768195.253243:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768195.253249:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066631936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768195.253275:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768195.253311:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768195.253324:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768195/real 1775768195] req@ffff9d81901ae680 x1862023066632064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768195.253336:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066632064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768195.253344:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768195.253347:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768195.253351:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768195.253356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768195.253375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768195.253387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768195/real 1775768195] req@ffff9d81901af100 x1862023066632192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768195.253398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066632192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768195.253405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768195.253408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768195.253410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768195.253416:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768195.253434:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768195.253444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768195/real 1775768195] req@ffff9d81901aed80 x1862023066632320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768195.253462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066632320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768195.253472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768195.253476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768195.253479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768195.253486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768195.253504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768195.253513:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768195/real 1775768195] req@ffff9d81901ad180 x1862023066632448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768195.253523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066632448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768195.253530:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768195.253533:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768195.253536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768195.893070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768196.276991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768196.917051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768197.300985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768197.941043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768198.324998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768198.965041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768199.348967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768199.989086:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768200.372968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768200.372983:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768200.372989:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768200.372993:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768200.373001:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5031 00000100:00080000:3.0:1775768200.373010:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768200.373016:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768200.373048:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066632576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768200.373069:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768200.373072:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768200.373074:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768200.373076:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5031 00000100:00080000:3.0:1775768200.373079:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768200.373081:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768200.373089:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768200.373098:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066632704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768200.373110:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768200.373114:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775768200.373116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066632576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768200.373116:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768200.373123:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5031 00000100:00080000:3.0:1775768200.373127:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768200.373129:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768200.373132:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768200.373141:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066632832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768200.373147:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768200.373150:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768200.373151:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768200.373154:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5031 00000100:00080000:3.0:1775768200.373156:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768200.373158:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768200.373161:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768200.373169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066632960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768200.373175:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768200.373178:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768200.373181:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775768200.373182:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768200.373185:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5031 00000100:00080000:3.0:1775768200.373189:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768200.373191:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768200.373194:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768200.373205:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066633088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768200.373230:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768200/real 1775768200] req@ffff9d8188741880 x1862023066632576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768200.373263:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066632576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768200.373272:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768200.373275:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768200.373278:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066632576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768200.373299:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768200.373322:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768200.373335:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768200/real 1775768200] req@ffff9d8188743800 x1862023066632704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768200.373349:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066632704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768200.373358:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768200.373361:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768200.373363:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768200.373368:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768200.373383:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768200.373391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768200/real 1775768200] req@ffff9d8188741f80 x1862023066632832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768200.373403:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066632832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768200.373418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768200.373421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768200.373425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768200.373431:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768200.373452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768200.373460:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768200/real 1775768200] req@ffff9d8188742a00 x1862023066632960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768200.373468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066632960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768200.373475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768200.373477:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768200.373479:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066632960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768200.373483:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768200.373494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768200.373500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768200/real 1775768200] req@ffff9d8188740700 x1862023066633088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768200.373506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066633088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768200.373512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768200.373513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768200.373516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768201.013044:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768201.396955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768202.037067:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768202.421008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768203.061088:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768203.445018:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768204.084997:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768204.468968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768205.109073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768205.492986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768205.493023:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768205.493031:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768205.493036:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768205.493046:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5037 00000100:00080000:3.0:1775768205.493057:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768205.493065:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768205.493107:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066633216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768205.493139:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768205.493143:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768205.493145:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768205.493149:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5037 00000100:00080000:3.0:1775768205.493152:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768205.493156:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768205.493166:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768205.493213:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066633344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768205.493226:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066633216:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768205.493226:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768205.493234:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768205.493237:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768205.493243:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5037 00000100:00080000:3.0:1775768205.493248:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768205.493253:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768205.493259:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768205.493273:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066633472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768205.493282:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768205.493286:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775768205.493287:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768205.493288:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768205.493294:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5037 00000100:00080000:3.0:1775768205.493298:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768205.493320:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768205.493327:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768205.493343:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066633600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768205.493354:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768205.493358:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768205.493361:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768205.493362:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768205/real 1775768205] req@ffff9d8188740380 x1862023066633216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768205.493369:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5037 00000100:00080000:3.0:1775768205.493376:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768205.493385:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768205.493390:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768205.493398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066633216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768205.493404:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066633728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768205.493431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768205.493438:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768205.493444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066633216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768205.493468:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768205.493491:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768205.493503:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768205/real 1775768205] req@ffff9d8188741c00 x1862023066633344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768205.493514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066633344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768205.493522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768205.493525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768205.493529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768205.493534:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768205.493550:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768205.493558:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768205/real 1775768205] req@ffff9d8188740000 x1862023066633472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768205.493569:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066633472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768205.493576:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768205.493578:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768205.493581:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768205.493586:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768205.493615:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768205.493621:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768205/real 1775768205] req@ffff9d8188740700 x1862023066633600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768205.493629:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066633600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768205.493643:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768205.493645:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768205.493647:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768205.493650:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768205.493662:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768205.493668:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768205/real 1775768205] req@ffff9d8188742a00 x1862023066633728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768205.493680:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066633728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768205.493686:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768205.493687:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768205.493689:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:0.0:1775768206.133070:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768206.516983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768207.157050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768207.540997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768208.181071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768208.564999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768209.205028:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768209.588984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768210.229100:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768210.613003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768210.613042:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768210.613050:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768210.613054:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768210.613063:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5042 00000100:00080000:2.0:1775768210.613074:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768210.613081:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768210.613120:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066633856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768210.613185:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768210.613188:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768210.613189:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768210.613192:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5042 00000100:00080000:2.0:1775768210.613196:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768210.613200:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768210.613208:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768210.613215:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066633856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768210.613223:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066633984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768210.613232:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768210.613235:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768210.613236:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768210.613239:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5042 00000100:00080000:2.0:1775768210.613242:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768210.613244:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768210.613248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768210.613255:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066634112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768210.613262:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768210.613264:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768210.613265:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768210.613270:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5042 00000800:00000100:1.0:1775768210.613271:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775768210.613275:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768210.613279:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768210.613283:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768210.613296:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066634240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768210.613305:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768210.613309:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768210.613311:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768210.613315:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5042 00000100:00080000:2.0:1775768210.613319:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768210.613324:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768210.613327:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768210/real 1775768210] req@ffff9d81984a2300 x1862023066633856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1775768210.613331:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768210.613348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066633856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768210.613351:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066634368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768210.613369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768210.613374:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768210.613378:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066633856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768210.613396:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768210.613412:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768210.613420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768210/real 1775768210] req@ffff9d81984a3b80 x1862023066633984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768210.613429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066633984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768210.613434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768210.613437:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768210.613439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066633984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768210.613442:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768210.613452:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768210.613465:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768210/real 1775768210] req@ffff9d81984a2d80 x1862023066634112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768210.613474:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066634112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768210.613479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768210.613481:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768210.613483:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768210.613487:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768210.613498:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768210.613504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768210/real 1775768210] req@ffff9d81984a3480 x1862023066634240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768210.613511:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066634240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768210.613522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768210.613525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768210.613527:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768210.613530:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768210.613539:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768210.613544:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768210/real 1775768210] req@ffff9d81984a2a00 x1862023066634368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768210.613551:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066634368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768210.613557:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768210.613558:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768210.613560:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768211.253017:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768211.637021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:0.0:1775768212.277054:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768212.660950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768213.300999:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768213.684945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768214.325020:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768214.708964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768215.349041:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768215.732979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768215.732998:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768215.733004:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768215.733009:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768215.733019:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5047 00000100:00080000:3.0:1775768215.733028:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768215.733034:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768215.733069:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066634496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768215.733110:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768215.733113:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768215.733115:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768215.733119:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5047 00000100:00080000:3.0:1775768215.733123:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768215.733127:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768215.733136:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768215.733145:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066634496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768215.733150:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066634624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768215.733161:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768215.733164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768215.733166:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768215.733171:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5047 00000100:00080000:3.0:1775768215.733174:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768215.733177:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768215.733181:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768215.733192:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066634752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768215.733201:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768215.733203:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768215.733207:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768215.733209:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768215.733214:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5047 00000100:00080000:3.0:1775768215.733220:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768215.733223:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768215.733228:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768215.733240:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066634880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768215.733250:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768215.733254:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768215/real 1775768215] req@ffff9d8188741f80 x1862023066634496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768215.733255:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768215.733258:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768215.733266:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5047 00000100:00080000:3.0:1775768215.733271:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768215.733275:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768215.733279:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768215.733282:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066634496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768215.733293:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1775768215.733293:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066635008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768215.733300:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768215.733309:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066634496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768215.733331:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768215.733350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768215.733360:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768215/real 1775768215] req@ffff9d8188743800 x1862023066634624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768215.733370:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066634624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768215.733377:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768215.733380:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768215.733382:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768215.733387:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768215.733401:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768215.733408:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768215/real 1775768215] req@ffff9d8188741880 x1862023066634752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768215.733417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066634752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768215.733424:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768215.733427:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768215.733430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768215.733435:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768215.733450:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768215.733458:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768215/real 1775768215] req@ffff9d8188742680 x1862023066634880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768215.733468:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066634880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768215.733477:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768215.733480:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768215.733483:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066634880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768215.733487:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768215.733502:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768215.733510:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768215/real 1775768215] req@ffff9d8188742300 x1862023066635008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768215.733520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066635008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768215.733527:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768215.733530:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768215.733532:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768216.373113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768216.756991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768217.397045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768217.780983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768218.421088:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768218.804983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768219.445047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768219.828967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768220.469026:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768220.852971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768220.852992:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768220.852998:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768220.853002:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768220.853011:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5052 00000100:00080000:2.0:1775768220.853021:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768220.853027:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768220.853095:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066635136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768220.853119:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768220.853124:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768220.853126:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768220.853129:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5052 00000100:00080000:2.0:1775768220.853133:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768220.853138:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768220.853146:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768220.853149:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066635136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768220.853160:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066635264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768220.853170:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768220.853173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768220.853175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768220.853178:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5052 00000100:00080000:2.0:1775768220.853195:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768220.853199:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768220.853203:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768220.853208:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775768220.853216:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066635392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768220.853225:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768220.853228:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768220.853230:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768220.853233:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5052 00000100:00080000:2.0:1775768220.853237:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768220.853240:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768220.853246:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768220.853257:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066635520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768220.853260:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768220/real 1775768220] req@ffff9d81984a0700 x1862023066635136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768220.853280:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768220.853284:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768220.853285:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768220.853289:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066635136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768220.853291:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5052 00000100:00080000:2.0:1775768220.853298:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768220.853304:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:1.0:1775768220.853305:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768220.853311:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:1.0:1775768220.853312:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768220.853318:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066635136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768220.853325:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066635648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768220.853340:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768220.853363:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768220.853373:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768220/real 1775768220] req@ffff9d81984a0000 x1862023066635264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768220.853381:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066635264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768220.853387:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768220.853389:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768220.853391:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768220.853395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768220.853413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768220.853419:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768220/real 1775768220] req@ffff9d81984a2680 x1862023066635392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768220.853426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066635392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768220.853431:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768220.853433:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768220.853435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768220.853438:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768220.853447:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768220.853452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768220/real 1775768220] req@ffff9d81984a0e00 x1862023066635520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768220.853458:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066635520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768220.853465:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768220.853466:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768220.853468:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768220.853471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768220.853479:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768220.853484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768220/real 1775768220] req@ffff9d81984a1f80 x1862023066635648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768220.853495:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066635648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768220.853499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768220.853501:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768220.853502:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768221.493046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768221.877046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768222.516984:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768222.900991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768223.541063:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768223.924981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768224.565050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768224.948984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768225.589069:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768225.972971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768225.973019:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768225.973026:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768225.973030:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768225.973042:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5057 00000100:00080000:3.0:1775768225.973052:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768225.973057:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768225.973087:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066635776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768225.973106:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768225.973108:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768225.973110:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768225.973112:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5057 00000100:00080000:3.0:1775768225.973114:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768225.973116:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768225.973123:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768225.973131:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066635904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768225.973140:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768225.973142:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768225.973143:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768225.973152:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5057 00000100:00100000:1.0:1775768225.973154:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066635776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768225.973155:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768225.973162:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768225.973173:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768225.973179:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066636032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768225.973191:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768225.973193:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768225.973194:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768225.973197:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5057 00000100:00080000:3.0:1775768225.973198:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768225.973200:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768225.973203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768225.973209:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066636160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768225.973214:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768225.973217:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768225.973218:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768225.973220:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5057 00000100:00080000:3.0:1775768225.973240:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768225.973242:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768225.973245:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768225.973253:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066636288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768225.973255:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768225.973322:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768225/real 1775768225] req@ffff9d8188740e00 x1862023066635776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768225.973341:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066635776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768225.973352:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768225.973356:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768225.973361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066635776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768225.973387:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768225.973408:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768225.973420:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768225/real 1775768225] req@ffff9d8188741180 x1862023066635904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768225.973432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066635904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768225.973440:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768225.973443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768225.973446:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066635904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768225.973452:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768225.973469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768225.973479:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768225/real 1775768225] req@ffff9d8188743100 x1862023066636032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768225.973490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066636032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768225.973497:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768225.973500:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768225.973503:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768225.973509:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768225.973527:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768225.973536:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768225/real 1775768225] req@ffff9d8188742d80 x1862023066636160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768225.973547:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066636160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768225.973562:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768225.973566:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768225.973568:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768225.973574:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768225.973591:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768225.973600:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768225/real 1775768225] req@ffff9d8188743b80 x1862023066636288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768225.973611:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066636288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768225.973619:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768225.973622:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768225.973625:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768226.613062:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768226.997038:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768227.637046:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768228.020934:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768228.661028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768229.045019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768229.685081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768230.069002:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768230.709120:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768231.092961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768231.093006:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768231.093011:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768231.093014:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768231.093021:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5062 00000100:00080000:1.0:1775768231.093029:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768231.093034:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768231.093066:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066636416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768231.093090:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768231.093092:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768231.093093:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768231.093096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5062 00000100:00080000:1.0:1775768231.093098:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768231.093102:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768231.093111:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768231.093122:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066636544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768231.093133:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066636416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768231.093133:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768231.093141:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768231.093144:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768231.093149:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5062 00000100:00080000:1.0:1775768231.093152:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768231.093156:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768231.093168:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768231.093180:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066636672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768231.093189:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768231.093193:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1775768231.093197:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768231.093197:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768231.093205:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5062 00000100:00080000:1.0:1775768231.093210:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768231.093214:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768231.093219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768231.093241:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066636800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768231.093248:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768231.093251:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775768231.093252:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768231/real 1775768231] req@ffff9d8190550e00 x1862023066636416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768231.093252:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768231.093258:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5062 00000100:00080000:1.0:1775768231.093263:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768231.093268:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768231.093272:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768231.093280:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066636928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768231.093283:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066636416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768231.093299:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768231.093304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768231.093308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066636416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768231.093327:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768231.093346:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768231.093356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768231/real 1775768231] req@ffff9d8190553100 x1862023066636544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768231.093372:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066636544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768231.093379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768231.093382:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768231.093384:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768231.093398:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768231.093414:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768231.093422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768231/real 1775768231] req@ffff9d8190551500 x1862023066636672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768231.093431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066636672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768231.093438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768231.093441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768231.093443:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768231.093452:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768231.093467:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768231.093475:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768231/real 1775768231] req@ffff9d8190551880 x1862023066636800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768231.093485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066636800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768231.093498:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768231.093501:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768231.093504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768231.093509:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768231.093522:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768231.093529:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768231/real 1775768231] req@ffff9d8190553480 x1862023066636928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768231.093538:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066636928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768231.093546:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768231.093548:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768231.093551:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066636928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768231.733107:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768232.117017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768232.757073:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768233.141046:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768233.781055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768234.165024:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768234.805112:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768235.189003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768235.829076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768236.213025:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768236.213028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768236.213044:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768236.213048:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768236.213054:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5067 00000100:00080000:0.0:1775768236.213062:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768236.213067:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768236.213099:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066637056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768236.213122:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768236.213126:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768236.213127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768236.213132:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5067 00000100:00080000:0.0:1775768236.213136:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768236.213141:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768236.213150:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768236.213154:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066637056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768236.213165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066637184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768236.213173:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768236.213175:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768236.213177:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768236.213180:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5067 00000100:00080000:0.0:1775768236.213182:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768236.213185:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768236.213188:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768236.213195:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066637312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768236.213201:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768236.213204:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768236.213205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768236.213208:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5067 00000100:00080000:0.0:1775768236.213212:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768236.213212:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775768236.213216:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768236.213219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768236.213226:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066637440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768236.213231:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768236.213233:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768236.213235:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768236.213237:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5067 00000100:00080000:0.0:1775768236.213239:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768236.213241:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768236.213243:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768236.213251:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066637568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768236.213269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768236/real 1775768236] req@ffff9d81901ad180 x1862023066637056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768236.213293:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066637056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768236.213302:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768236.213306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768236.213310:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066637056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768236.213330:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768236.213352:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768236.213364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768236/real 1775768236] req@ffff9d81901aed80 x1862023066637184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768236.213376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066637184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768236.213384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768236.213387:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768236.213389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768236.213393:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768236.213415:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768236.213422:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768236/real 1775768236] req@ffff9d81901af100 x1862023066637312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768236.213429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066637312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768236.213437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768236.213439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768236.213441:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768236.213444:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768236.213454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768236.213459:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768236/real 1775768236] req@ffff9d81901ae680 x1862023066637440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768236.213466:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066637440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768236.213475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768236.213478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768236.213481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768236.213486:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768236.213503:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768236.213512:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768236/real 1775768236] req@ffff9d81901afb80 x1862023066637568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768236.213522:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066637568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768236.213529:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768236.213531:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768236.213535:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768236.853071:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768237.237019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768237.877088:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768238.260985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768238.901120:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768239.285009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768239.925017:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768240.308994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768240.949047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768241.332970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768241.332976:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768241.332982:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768241.332985:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768241.332992:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5072 00000100:00080000:2.0:1775768241.333002:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768241.333009:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768241.333043:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066637696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768241.333068:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768241.333071:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768241.333072:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768241.333075:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5072 00000100:00080000:2.0:1775768241.333080:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768241.333084:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768241.333094:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768241.333101:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066637696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768241.333108:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066637824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768241.333119:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768241.333123:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768241.333125:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768241.333129:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5072 00000100:00080000:2.0:1775768241.333132:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768241.333134:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768241.333138:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768241.333145:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066637952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768241.333151:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768241.333153:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768241.333154:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768241.333158:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5072 00000800:00000100:1.0:1775768241.333160:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775768241.333162:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768241.333165:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768241.333168:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768241.333174:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066638080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768241.333184:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768241.333186:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768241.333187:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768241.333189:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5072 00000100:00080000:2.0:1775768241.333191:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768241.333194:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768241.333196:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775768241.333203:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066638208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768241.333210:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768241/real 1775768241] req@ffff9d81984a1180 x1862023066637696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768241.333229:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066637696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768241.333238:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768241.333242:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768241.333246:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066637696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768241.333265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768241.333286:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768241.333298:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768241/real 1775768241] req@ffff9d81984a1880 x1862023066637824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768241.333309:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066637824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768241.333317:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768241.333320:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768241.333323:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768241.333329:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768241.333347:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768241.333357:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768241/real 1775768241] req@ffff9d81984a3800 x1862023066637952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768241.333368:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066637952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768241.333375:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768241.333378:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768241.333381:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066637952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768241.333387:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768241.333405:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768241.333415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768241/real 1775768241] req@ffff9d81984a3100 x1862023066638080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768241.333435:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066638080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768241.333447:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768241.333451:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768241.333453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768241.333459:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768241.333476:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768241.333486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768241/real 1775768241] req@ffff9d81984a1500 x1862023066638208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768241.333496:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066638208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768241.333504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768241.333507:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768241.333510:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768241.973139:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768242.356992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768242.997032:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768243.380968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768244.021075:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768244.404996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768245.045070:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768245.428947:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768246.069050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768246.452988:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768246.452990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768246.452996:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768246.453000:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768246.453008:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5077 00000100:00080000:3.0:1775768246.453017:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768246.453022:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768246.453057:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066638336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768246.453081:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768246.453085:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768246.453087:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768246.453090:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5077 00000100:00080000:3.0:1775768246.453093:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768246.453098:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768246.453114:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768246.453119:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066638336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768246.453133:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066638464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768246.453144:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768246.453148:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768246.453150:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768246.453154:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5077 00000100:00080000:3.0:1775768246.453171:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768246.453175:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768246.453180:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768246.453181:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768246.453197:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066638592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768246.453204:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768246.453206:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768246.453207:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768246.453210:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5077 00000100:00080000:3.0:1775768246.453212:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768246.453215:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768246.453219:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768246.453226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066638720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768246.453232:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768246.453234:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768246.453235:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768246.453237:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5077 00000100:00080000:3.0:1775768246.453239:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768246.453240:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768246.453243:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768246.453251:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066638848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768246.453258:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768246/real 1775768246] req@ffff9d8188743480 x1862023066638336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768246.453279:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066638336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768246.453290:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768246.453294:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768246.453298:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066638336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768246.453320:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768246.453342:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768246.453362:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768246/real 1775768246] req@ffff9d8188740a80 x1862023066638464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768246.453374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066638464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768246.453390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768246.453394:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768246.453397:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768246.453404:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768246.453424:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768246.453435:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768246/real 1775768246] req@ffff9d8188741500 x1862023066638592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768246.453447:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066638592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768246.453454:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768246.453457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768246.453460:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768246.453466:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768246.453486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768246.453496:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768246/real 1775768246] req@ffff9d8188743b80 x1862023066638720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768246.453508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066638720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768246.453519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768246.453522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768246.453525:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768246.453531:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768246.453549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768246.453559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768246/real 1775768246] req@ffff9d8188742d80 x1862023066638848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768246.453570:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066638848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768246.453578:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768246.453582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768246.453585:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066638848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768247.093092:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768247.477028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768248.117124:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768248.501015:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768249.141108:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768249.525067:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768250.165065:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768250.548998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768251.189046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768251.572981:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768251.572987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768251.572991:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768251.572995:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768251.573005:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5083 00000100:00080000:3.0:1775768251.573016:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768251.573024:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768251.573060:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066638976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768251.573088:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768251.573092:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768251.573093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768251.573097:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5083 00000100:00080000:3.0:1775768251.573101:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768251.573104:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768251.573113:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768251.573125:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066639104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768251.573135:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768251.573139:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775768251.573141:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066638976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768251.573141:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768251.573148:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5083 00000100:00080000:3.0:1775768251.573151:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768251.573154:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768251.573159:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768251.573169:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066639232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768251.573178:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768251.573181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768251.573183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768251.573187:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5083 00000100:00080000:3.0:1775768251.573190:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768251.573196:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768251.573206:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768251.573215:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768251.573231:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066639360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768251.573237:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768251/real 1775768251] req@ffff9d8188743100 x1862023066638976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768251.573267:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768251.573270:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768251.573272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768251.573276:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066638976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768251.573278:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5083 00000100:00080000:3.0:1775768251.573281:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768251.573288:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768251.573296:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:1.0:1775768251.573300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768251.573306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:3.0:1775768251.573309:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066639488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768251.573314:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066638976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768251.573344:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768251.573367:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768251.573379:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768251/real 1775768251] req@ffff9d8188741180 x1862023066639104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768251.573398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066639104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768251.573407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768251.573410:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768251.573413:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768251.573420:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768251.573439:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768251.573447:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768251/real 1775768251] req@ffff9d8188740e00 x1862023066639232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768251.573454:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066639232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768251.573459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768251.573461:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768251.573463:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768251.573467:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768251.573478:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768251.573483:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768251/real 1775768251] req@ffff9d8188742300 x1862023066639360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768251.573490:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066639360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768251.573495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768251.573497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768251.573499:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768251.573501:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768251.573510:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768251.573514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768251/real 1775768251] req@ffff9d8188742680 x1862023066639488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768251.573520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066639488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768251.573525:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768251.573526:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768251.573528:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768252.213027:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768252.596978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768253.237053:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768253.620992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768254.261046:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768254.644980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768255.285048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768255.668949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768256.309032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768256.692980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768256.692982:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768256.692988:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768256.692991:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768256.692998:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5088 00000100:00080000:1.0:1775768256.693007:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768256.693014:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768256.693051:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066639616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768256.693079:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768256.693081:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768256.693083:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768256.693086:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5088 00000100:00080000:1.0:1775768256.693088:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768256.693091:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768256.693097:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768256.693107:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066639744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768256.693114:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768256.693117:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768256.693118:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768256.693123:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066639616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768256.693124:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5088 00000100:00080000:1.0:1775768256.693128:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768256.693131:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768256.693133:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768256.693141:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066639872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768256.693147:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768256.693149:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768256.693150:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768256.693153:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5088 00000100:00080000:1.0:1775768256.693167:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768256.693169:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768256.693173:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768256.693179:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066640000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768256.693185:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768256.693187:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768256.693188:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768256.693190:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5088 00000100:00080000:1.0:1775768256.693191:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768256.693193:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768256.693196:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768256.693204:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066640128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768256.693217:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768256.693241:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768256.693253:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768256.693265:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768257.333049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768257.717075:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768258.357055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768258.741080:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768259.381049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768259.765093:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:1.0:1775768259.765208:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:1.0:1775768259.765243:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1775768259.765257:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775768259.765295:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775768259.765303:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775768259.765311:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:1.0:1775768259.765317:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:0.0:1775768259.765395:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768256/real 1775768259] req@ffff9d8190551880 x1862023066639744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768259.765423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066639744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768259.765443:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768259.765449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768259.765453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768259.765461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768256/real 1775768259] req@ffff9d8190551500 x1862023066639872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768259.765472:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066639872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768259.765479:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768259.765483:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768259.765486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066639872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768259.765492:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768256/real 1775768259] req@ffff9d8190553100 x1862023066640000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768259.765502:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066640000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768259.765509:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768259.765513:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768259.765515:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768259.765521:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768256/real 1775768259] req@ffff9d8190550e00 x1862023066640128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768259.765531:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066640128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768259.765538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768259.765542:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768259.765545:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640128:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768259.765550:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768256/real 1775768259] req@ffff9d8190553480 x1862023066639616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768259.765561:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066639616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768259.765567:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768259.765570:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768259.765573:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066639616:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775768260.405045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768260.789048:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768261.429064:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768261.812973:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768261.812983:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768261.812987:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768261.812997:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5093 00000100:00080000:1.0:1775768261.813006:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768261.813012:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768261.813031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:1.0:1775768261.813047:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066640256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768261.813075:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768261.813077:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768261.813078:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768261.813081:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5093 00000100:00080000:1.0:1775768261.813084:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768261.813087:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768261.813095:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768261.813107:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066640384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768261.813116:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066640256:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768261.813118:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768261.813124:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768261.813126:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768261.813130:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5093 00000100:00080000:1.0:1775768261.813133:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768261.813136:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768261.813140:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768261.813165:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066640512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768261.813181:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768261.813184:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768261.813186:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768261.813188:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768261.813193:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5093 00000100:00080000:1.0:1775768261.813198:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768261.813202:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768261.813207:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768261.813227:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066640640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768261.813229:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768261/real 1775768261] req@ffff9d8190553480 x1862023066640256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768261.813246:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768261.813251:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768261.813252:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768261.813257:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5093 00000100:00100000:0.0:1775768261.813260:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066640256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768261.813262:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768261.813270:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768261.813277:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:0.0:1775768261.813279:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768261.813286:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768261.813293:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066640256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768261.813293:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066640768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768261.813333:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768261.813359:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768261.813371:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768261/real 1775768261] req@ffff9d8190550e00 x1862023066640384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768261.813383:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066640384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768261.813390:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768261.813393:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768261.813396:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768261.813402:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768261.813419:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768261.813428:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768261/real 1775768261] req@ffff9d8190553100 x1862023066640512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768261.813440:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066640512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768261.813448:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768261.813450:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768261.813453:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768261.813459:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768261.813477:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768261.813486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768261/real 1775768261] req@ffff9d8190551500 x1862023066640640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768261.813503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066640640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768261.813512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768261.813515:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768261.813518:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768261.813522:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768261.813540:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768261.813548:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768261/real 1775768261] req@ffff9d8190551880 x1862023066640768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768261.813558:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066640768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768261.813565:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768261.813568:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768261.813571:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066640768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768262.453056:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768262.837044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768263.477006:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768263.860967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768264.501037:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768264.884977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768265.525028:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768265.908985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768266.549054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768266.932945:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768266.932996:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768266.933004:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768266.933008:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768266.933016:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5098 00000100:00080000:3.0:1775768266.933026:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768266.933032:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768266.933064:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066640896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768266.933091:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768266.933094:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768266.933101:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768266.933104:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5098 00000100:00080000:3.0:1775768266.933108:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768266.933111:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768266.933119:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768266.933131:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066640896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768266.933132:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066641024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768266.933146:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768266.933149:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768266.933151:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768266.933155:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5098 00000100:00080000:3.0:1775768266.933159:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768266.933162:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768266.933167:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768266.933177:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066641152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768266.933182:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768266.933196:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768266.933199:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768266.933201:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768266.933206:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5098 00000100:00080000:3.0:1775768266.933210:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768266.933213:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768266.933218:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768266/real 1775768266] req@ffff9d8188741880 x1862023066640896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1775768266.933218:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768266.933234:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066640896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768266.933237:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066641280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768266.933267:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768266.933268:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768266.933274:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775768266.933275:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1775768266.933278:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066640896:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768266.933278:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768266.933286:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5098 00000100:00080000:3.0:1775768266.933291:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768266.933294:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768266.933300:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641024:192.168.202.105@tcp:38:kworker.0 00000008:00100000:3.0:1775768266.933300:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768266.933317:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066641408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768266.933321:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768266.933334:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768266/real 1775768266] req@ffff9d8188743800 x1862023066641024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768266.933348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066641024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768266.933356:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768266.933360:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768266.933363:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768266.933369:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768266.933385:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768266.933393:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768266/real 1775768266] req@ffff9d8188741f80 x1862023066641152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768266.933400:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066641152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768266.933405:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768266.933407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768266.933409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768266.933412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768266.933422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768266.933427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768266/real 1775768266] req@ffff9d8188742a00 x1862023066641280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768266.933433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066641280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768266.933439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768266.933441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768266.933442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768266.933451:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768266.933461:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768266.933469:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768266/real 1775768266] req@ffff9d8188740700 x1862023066641408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768266.933480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066641408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768266.933488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768266.933491:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768266.933494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768267.573044:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768267.956989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768268.597056:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768268.980988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768269.621006:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768270.004993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768270.645049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768271.028955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768271.669001:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768272.052980:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768272.052983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768272.052989:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768272.052993:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768272.053000:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5103 00000100:00080000:3.0:1775768272.053008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768272.053013:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768272.053045:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066641536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768272.053071:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768272.053075:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768272.053076:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768272.053080:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5103 00000100:00080000:3.0:1775768272.053083:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768272.053087:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768272.053096:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768272.053109:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066641536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768272.053112:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066641664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768272.053129:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768272.053132:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768272.053133:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768272.053137:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5103 00000100:00080000:3.0:1775768272.053153:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768272.053155:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768272.053159:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775768272.053163:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768272.053172:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066641792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768272.053179:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768272.053182:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768272.053183:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768272.053185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5103 00000100:00080000:3.0:1775768272.053187:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768272.053189:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768272.053192:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768272.053201:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066641920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768272.053206:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768272.053208:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768272.053209:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768272.053212:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5103 00000100:00080000:3.0:1775768272.053215:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768272.053218:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768272.053223:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768272.053227:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768272/real 1775768272] req@ffff9d8188740000 x1862023066641536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768272.053236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066642048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768272.053252:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066641536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768272.053260:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768272.053264:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768272.053267:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066641536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768272.053284:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641664:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768272.053300:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768272.053308:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768272/real 1775768272] req@ffff9d8188741c00 x1862023066641664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768272.053315:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066641664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768272.053320:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768272.053321:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768272.053323:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768272.053326:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768272.053335:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768272.053339:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768272/real 1775768272] req@ffff9d8188740380 x1862023066641792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768272.053345:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066641792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768272.053354:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768272.053355:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768272.053357:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768272.053360:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768272.053368:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768272.053373:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768272/real 1775768272] req@ffff9d8188740700 x1862023066641920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768272.053378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066641920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768272.053384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768272.053386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768272.053387:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066641920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768272.053390:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768272.053398:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768272.053402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768272/real 1775768272] req@ffff9d8188742a00 x1862023066642048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768272.053408:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066642048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768272.053412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768272.053414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768272.053416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768272.693036:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768273.076960:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768273.717053:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768274.100982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768274.741053:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768275.125000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768275.765051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768276.148976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768276.789006:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768277.172953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768277.172955:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768277.172963:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768277.172968:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768277.172978:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5108 00000100:00080000:3.0:1775768277.172988:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768277.172995:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768277.173030:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066642176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768277.173051:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768277.173053:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768277.173054:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768277.173057:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5108 00000100:00080000:3.0:1775768277.173059:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768277.173062:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768277.173069:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768277.173077:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066642304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768277.173084:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768277.173086:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768277.173088:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768277.173090:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5108 00000100:00080000:3.0:1775768277.173092:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768277.173094:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768277.173097:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768277.173101:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066642176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768277.173106:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066642432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768277.173115:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768277.173117:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768277.173118:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768277.173120:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5108 00000100:00080000:3.0:1775768277.173122:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768277.173124:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768277.173126:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768277.173134:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066642560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768277.173140:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768277.173142:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768277.173143:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768277.173145:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5108 00000100:00080000:3.0:1775768277.173153:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768277.173155:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768277.173158:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000800:00000100:0.0:1775768277.173162:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768277.173170:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066642688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768277.173206:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768277/real 1775768277] req@ffff9d8188741f80 x1862023066642176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768277.173221:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066642176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768277.173230:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768277.173234:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768277.173237:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066642176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768277.173244:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768277.173264:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768277.173274:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768277/real 1775768277] req@ffff9d8188743800 x1862023066642304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768277.173285:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066642304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768277.173293:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768277.173296:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768277.173299:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768277.173319:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768277.173334:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768277.173343:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768277/real 1775768277] req@ffff9d8188741880 x1862023066642432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768277.173353:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066642432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768277.173361:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768277.173363:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768277.173367:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768277.173372:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768277.173386:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768277.173394:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768277/real 1775768277] req@ffff9d8188742680 x1862023066642560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768277.173403:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066642560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768277.173412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768277.173415:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768277.173417:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768277.173422:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768277.173436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768277.173444:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768277/real 1775768277] req@ffff9d8188742300 x1862023066642688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768277.173453:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066642688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768277.173461:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768277.173464:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768277.173467:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768277.813079:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768278.197010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768278.837034:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768279.220959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768279.861103:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768280.244953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768280.884998:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768281.268994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768281.909034:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768282.292979:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768282.292988:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1775768282.292992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768282.292992:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768282.293005:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5113 00000100:00080000:3.0:1775768282.293015:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768282.293023:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768282.293058:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066642816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768282.293086:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768282.293090:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768282.293091:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768282.293095:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5113 00000100:00080000:3.0:1775768282.293099:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768282.293102:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768282.293117:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768282.293132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066642816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768282.293132:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066642944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768282.293147:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768282.293151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768282.293152:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768282.293156:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5113 00000100:00080000:3.0:1775768282.293160:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768282.293165:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768282.293170:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768282.293182:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066643072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768282.293186:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768282.293195:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768282.293198:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768282.293200:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768282.293205:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5113 00000100:00080000:3.0:1775768282.293209:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768282.293213:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768282.293219:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768282.293221:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768282/real 1775768282] req@ffff9d8188740e00 x1862023066642816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768282.293233:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066643200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768282.293237:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066642816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768282.293255:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768282.293256:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768282.293260:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768282.293262:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775768282.293264:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768282.293267:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066642816:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768282.293272:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5113 00000100:00080000:3.0:1775768282.293276:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768282.293280:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768282.293285:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768282.293288:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775768282.293301:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066643328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768282.293308:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768282.293319:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768282/real 1775768282] req@ffff9d8188741180 x1862023066642944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768282.293341:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066642944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768282.293349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768282.293353:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768282.293356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066642944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768282.293362:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768282.293381:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768282.293391:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768282/real 1775768282] req@ffff9d8188743100 x1862023066643072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768282.293411:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066643072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768282.293418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768282.293421:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768282.293424:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768282.293429:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768282.293462:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768282.293477:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768282/real 1775768282] req@ffff9d8188742d80 x1862023066643200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768282.293488:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066643200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768282.293497:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768282.293500:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768282.293502:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768282.293513:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768282.293526:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768282.293533:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768282/real 1775768282] req@ffff9d8188743b80 x1862023066643328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768282.293539:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066643328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768282.293544:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768282.293546:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768282.293548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768282.933077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768283.316999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768283.957046:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768284.340985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768284.981096:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768285.365036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768286.005418:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768286.389020:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768287.029060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768287.412971:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768287.412981:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768287.412986:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768287.412994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768287.412996:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5118 00000100:00080000:3.0:1775768287.413008:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768287.413015:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768287.413065:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066643456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768287.413099:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768287.413102:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768287.413103:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768287.413106:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5118 00000100:00080000:3.0:1775768287.413109:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768287.413112:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768287.413121:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768287.413132:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066643584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768287.413139:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768287.413141:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768287.413143:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768287.413146:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5118 00000100:00080000:3.0:1775768287.413150:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768287.413152:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066643456:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768287.413155:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768287.413161:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768287.413167:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066643712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768287.413174:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768287.413177:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768287.413178:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768287.413180:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5118 00000100:00080000:3.0:1775768287.413182:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768287.413184:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768287.413187:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768287.413202:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066643840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768287.413210:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768287.413212:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768287.413214:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768287.413216:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5118 00000100:00080000:3.0:1775768287.413219:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768287.413221:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768287.413225:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775768287.413226:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768287.413235:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066643968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768287.413271:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768287/real 1775768287] req@ffff9d8188741500 x1862023066643456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768287.413286:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066643456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768287.413300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768287.413305:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768287.413309:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066643456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768287.413316:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768287.413339:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768287.413350:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768287/real 1775768287] req@ffff9d8188740a80 x1862023066643584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768287.413362:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066643584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768287.413372:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768287.413374:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768287.413376:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768287.413417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768287.413432:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768287.413439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768287/real 1775768287] req@ffff9d8188743480 x1862023066643712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768287.413456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066643712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768287.413464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768287.413467:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768287.413469:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768287.413475:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768287.413494:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768287.413504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768287/real 1775768287] req@ffff9d8188743b80 x1862023066643840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768287.413515:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066643840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768287.413522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768287.413524:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768287.413526:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768287.413531:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768287.413549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768287.413559:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768287/real 1775768287] req@ffff9d8188742d80 x1862023066643968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768287.413570:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066643968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768287.413579:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768287.413582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768287.413585:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066643968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768288.053104:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768288.437017:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768289.077022:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768289.461011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768290.101037:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768290.485012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768291.125044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768291.508969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768292.149040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768292.532977:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768292.532996:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768292.533005:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768292.533009:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768292.533028:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5124 00000100:00080000:3.0:1775768292.533041:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768292.533049:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768292.533096:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066644096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768292.533122:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768292.533125:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768292.533126:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768292.533129:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5124 00000100:00080000:3.0:1775768292.533132:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768292.533135:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768292.533145:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768292.533154:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066644224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768292.533165:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768292.533168:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066644096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768292.533171:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768292.533174:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768292.533180:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5124 00000100:00080000:3.0:1775768292.533183:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768292.533188:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768292.533193:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768292.533205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066644352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768292.533213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768292.533224:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768292.533225:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768292.533230:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5124 00000100:00080000:3.0:1775768292.533234:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768292.533238:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768292.533243:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:3.0:1775768292.533246:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768292.533260:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066644480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768292.533267:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768292.533269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768292.533271:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768292.533273:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5124 00000100:00080000:3.0:1775768292.533276:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768292.533279:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768292.533282:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768292.533293:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768292/real 1775768292] req@ffff9d8188743100 x1862023066644096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768292.533295:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066644608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768292.533323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066644096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768292.533331:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768292.533335:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768292.533338:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066644096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768292.533374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768292.533393:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768292.533405:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768292/real 1775768292] req@ffff9d8188741180 x1862023066644224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768292.533416:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066644224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768292.533423:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768292.533426:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768292.533430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768292.533436:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768292.533454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768292.533464:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768292/real 1775768292] req@ffff9d8188740e00 x1862023066644352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768292.533474:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066644352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768292.533480:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768292.533483:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768292.533486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768292.533491:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768292.533509:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768292.533519:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768292/real 1775768292] req@ffff9d8188742300 x1862023066644480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768292.533529:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066644480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768292.533538:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768292.533541:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768292.533544:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768292.533548:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768292.533562:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768292.533568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768292/real 1775768292] req@ffff9d8188742680 x1862023066644608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768292.533575:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066644608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768292.533580:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768292.533582:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768292.533584:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768293.173050:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768293.556981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768294.197027:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768294.580991:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768295.221062:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768295.604978:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768296.244990:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768296.628936:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768297.269065:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768297.652954:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768297.652961:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768297.652964:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768297.652969:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768297.652978:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5129 00000100:00080000:3.0:1775768297.652988:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768297.652996:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768297.653026:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066644736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768297.653051:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768297.653055:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768297.653056:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768297.653060:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5129 00000100:00080000:3.0:1775768297.653064:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768297.653068:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768297.653076:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768297.653088:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066644864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768297.653099:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066644736:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768297.653101:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768297.653114:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768297.653116:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768297.653120:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5129 00000100:00080000:3.0:1775768297.653124:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768297.653128:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768297.653133:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768297.653142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066644992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768297.653151:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768297.653154:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768297.653155:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775768297.653160:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768297.653163:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5129 00000100:00080000:3.0:1775768297.653167:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768297.653171:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768297.653175:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768297.653186:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066645120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768297.653195:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768297.653198:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768297.653199:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768297.653203:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5129 00000100:00100000:0.0:1775768297.653207:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768297/real 1775768297] req@ffff9d8188741880 x1862023066644736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768297.653207:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768297.653213:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768297.653221:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768297.653230:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066645248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768297.653249:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066644736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768297.653260:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768297.653264:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768297.653268:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066644736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768297.653289:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768297.653337:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768297.653349:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768297/real 1775768297] req@ffff9d8188743800 x1862023066644864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768297.653360:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066644864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768297.653369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768297.653372:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768297.653375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768297.653381:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768297.653394:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768297.653401:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768297/real 1775768297] req@ffff9d8188741f80 x1862023066644992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768297.653407:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066644992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768297.653412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768297.653414:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768297.653415:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066644992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768297.653418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768297.653432:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768297.653440:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768297/real 1775768297] req@ffff9d8188742a00 x1862023066645120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768297.653451:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066645120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768297.653460:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768297.653462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768297.653465:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768297.653471:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768297.653486:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768297.653495:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768297/real 1775768297] req@ffff9d8188740700 x1862023066645248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768297.653506:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066645248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768297.653514:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768297.653517:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768297.653520:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768298.292990:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768298.676938:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768299.317032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768299.700973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768300.341048:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768300.724997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768301.365003:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768301.748953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768302.389042:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768302.772973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768302.772976:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768302.772982:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768302.772985:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768302.772992:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5134 00000100:00080000:1.0:1775768302.773001:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768302.773006:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768302.773036:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066645376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768302.773060:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768302.773062:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768302.773063:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768302.773068:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5134 00000100:00080000:1.0:1775768302.773070:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768302.773072:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768302.773079:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768302.773087:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066645504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768302.773094:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768302.773096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768302.773098:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768302.773101:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066645376:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768302.773103:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5134 00000100:00080000:1.0:1775768302.773108:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768302.773110:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768302.773113:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768302.773119:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066645632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768302.773125:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768302.773127:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768302.773129:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768302.773132:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5134 00000100:00080000:1.0:1775768302.773144:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768302.773148:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768302.773153:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775768302.773155:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768302.773169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066645760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768302.773178:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768302.773181:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768302.773183:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768302.773188:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5134 00000100:00100000:0.0:1775768302.773189:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768302/real 1775768302] req@ffff9d8190551880 x1862023066645376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768302.773191:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768302.773197:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768302.773203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768302.773209:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066645376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768302.773214:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066645888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768302.773225:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768302.773229:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768302.773233:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066645376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768302.773260:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768302.773276:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768302.773284:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768302/real 1775768302] req@ffff9d8190551500 x1862023066645504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768302.773292:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066645504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768302.773297:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768302.773299:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768302.773301:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768302.773304:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768302.773313:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768302.773317:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768302/real 1775768302] req@ffff9d8190553100 x1862023066645632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768302.773323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066645632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768302.773328:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768302.773330:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768302.773332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768302.773335:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768302.773343:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768302.773348:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768302/real 1775768302] req@ffff9d8190550e00 x1862023066645760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768302.773354:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066645760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768302.773360:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768302.773361:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768302.773362:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768302.773365:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768302.773374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768302.773378:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768302/real 1775768302] req@ffff9d8190553480 x1862023066645888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768302.773384:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066645888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768302.773389:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768302.773390:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768302.773392:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066645888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768303.412992:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768303.796946:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768304.437009:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768304.820993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768305.461036:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768305.844966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768306.485004:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768306.868959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768307.509064:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768307.892942:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768307.892949:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768307.892953:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768307.892961:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5139 00000100:00100000:2.0:1775768307.892967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768307.892969:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768307.892975:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768307.893007:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066646016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768307.893032:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768307.893036:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768307.893037:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768307.893041:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5139 00000100:00080000:1.0:1775768307.893045:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768307.893049:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768307.893059:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768307.893067:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066646016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768307.893074:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066646144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768307.893086:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768307.893089:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768307.893091:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768307.893096:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5139 00000100:00080000:1.0:1775768307.893108:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768307.893114:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768307.893115:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:1.0:1775768307.893123:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768307.893137:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066646272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768307.893147:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768307.893151:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768307/real 1775768307] req@ffff9d8190553480 x1862023066646016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768307.893152:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768307.893156:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768307.893165:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5139 00000100:00080000:1.0:1775768307.893170:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768307.893172:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066646016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768307.893177:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:0.0:1775768307.893183:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1775768307.893184:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00080000:0.0:1775768307.893188:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768307.893191:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066646016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768307.893207:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066646400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768307.893214:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768307.893216:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768307.893217:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768307.893220:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5139 00000100:00080000:1.0:1775768307.893223:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768307.893224:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646144:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775768307.893228:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768307.893233:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768307.893242:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066646528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768307.893248:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768307.893263:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768307/real 1775768307] req@ffff9d8190550e00 x1862023066646144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768307.893270:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066646144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768307.893275:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768307.893277:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768307.893279:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768307.893282:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768307.893291:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768307.893296:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768307/real 1775768307] req@ffff9d8190553100 x1862023066646272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768307.893302:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066646272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768307.893310:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768307.893312:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768307.893314:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768307.893317:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768307.893325:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768307.893329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768307/real 1775768307] req@ffff9d8190551500 x1862023066646400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768307.893335:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066646400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768307.893342:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768307.893344:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768307.893346:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768307.893356:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768307.893364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768307.893369:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768307/real 1775768307] req@ffff9d8190551880 x1862023066646528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768307.893375:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066646528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768307.893379:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768307.893381:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768307.893383:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768308.533047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768308.916976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768309.557051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768309.940987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768310.580995:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768310.964990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768311.605043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768311.988969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768312.629055:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768313.012974:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768313.012988:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768313.012996:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768313.013000:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768313.013009:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5144 00000100:00080000:0.0:1775768313.013017:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768313.013022:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768313.013053:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066646656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768313.013073:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768313.013075:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768313.013076:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768313.013079:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5144 00000100:00080000:0.0:1775768313.013081:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768313.013083:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768313.013089:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768313.013097:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066646784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768313.013104:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768313.013108:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768313.013109:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768313.013109:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066646656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768313.013116:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5144 00000100:00080000:0.0:1775768313.013120:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768313.013122:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768313.013125:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768313.013132:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066646912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768313.013137:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768313.013139:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768313.013140:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768313.013142:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5144 00000100:00080000:0.0:1775768313.013157:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768313.013160:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768313.013163:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768313.013170:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066647040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768313.013176:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768313.013178:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768313.013179:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775768313.013184:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775768313.013185:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5144 00000100:00080000:0.0:1775768313.013189:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768313.013191:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768313.013194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768313.013201:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066647168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768313.013232:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768313/real 1775768313] req@ffff9d81901afb80 x1862023066646656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768313.013249:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066646656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768313.013264:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768313.013269:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768313.013273:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066646656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768313.013294:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768313.013315:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768313.013326:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768313/real 1775768313] req@ffff9d81901ae680 x1862023066646784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768313.013338:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066646784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768313.013346:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768313.013349:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768313.013352:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768313.013358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768313.013375:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768313.013384:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768313/real 1775768313] req@ffff9d81901af100 x1862023066646912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768313.013395:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066646912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768313.013403:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768313.013406:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768313.013409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066646912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768313.013415:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768313.013430:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768313.013439:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768313/real 1775768313] req@ffff9d81901aed80 x1862023066647040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768313.013450:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066647040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768313.013459:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768313.013462:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768313.013465:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768313.013470:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768313.013485:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768313.013493:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768313/real 1775768313] req@ffff9d81901ad180 x1862023066647168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768313.013503:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066647168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768313.013511:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768313.013514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768313.013516:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768313.653019:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768314.036964:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768314.677016:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768315.060976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768315.701023:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768316.084950:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768316.725077:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768317.109019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768317.749072:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768318.132967:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768318.132974:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768318.132977:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768318.132987:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5149 00000100:00100000:2.0:1775768318.132993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768318.132996:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768318.133005:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768318.133035:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066647296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768318.133076:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768318.133079:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768318.133081:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768318.133084:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5149 00000100:00080000:3.0:1775768318.133087:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768318.133091:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768318.133110:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768318.133124:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066647424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768318.133127:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066647296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768318.133141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768318.133144:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768318.133146:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768318.133151:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5149 00000100:00080000:3.0:1775768318.133154:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768318.133158:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768318.133163:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768318.133173:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066647552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768318.133181:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768318.133184:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768318.133186:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1775768318.133189:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768318.133194:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5149 00000100:00080000:3.0:1775768318.133198:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768318.133202:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768318.133207:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768318.133219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066647680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768318.133227:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768318.133230:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768318.133231:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768318.133235:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5149 00000100:00080000:3.0:1775768318.133238:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768318.133241:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768318/real 1775768318] req@ffff9d8188740380 x1862023066647296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768318.133245:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768318.133251:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768318.133271:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066647296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768318.133272:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066647808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768318.133301:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768318.133307:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768318.133312:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066647296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768318.133331:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768318.133348:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768318.133356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768318/real 1775768318] req@ffff9d8188741c00 x1862023066647424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768318.133363:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066647424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768318.133368:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768318.133371:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768318.133373:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768318.133379:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768318.133396:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768318.133406:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768318/real 1775768318] req@ffff9d8188740000 x1862023066647552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768318.133417:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066647552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768318.133425:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768318.133428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768318.133431:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768318.133437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768318.133456:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768318.133466:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768318/real 1775768318] req@ffff9d8188740700 x1862023066647680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768318.133475:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066647680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768318.133481:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768318.133483:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768318.133484:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768318.133488:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768318.133500:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768318.133505:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768318/real 1775768318] req@ffff9d8188742a00 x1862023066647808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768318.133512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066647808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768318.133517:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768318.133518:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768318.133520:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066647808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768318.773054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768319.156972:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768319.797018:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768320.181001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768320.821043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768321.204999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768321.845033:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768322.228993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768322.869092:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768323.252962:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768323.252970:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768323.252971:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768323.252977:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768323.252987:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5154 00000100:00080000:3.0:1775768323.252997:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768323.253005:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768323.253039:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066647936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768323.253078:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768323.253082:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768323.253084:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768323.253087:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5154 00000100:00080000:3.0:1775768323.253091:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768323.253094:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768323.253103:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768323.253117:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066647936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768323.253117:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066648064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768323.253132:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768323.253135:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768323.253136:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768323.253140:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5154 00000100:00080000:3.0:1775768323.253144:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768323.253147:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768323.253153:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768323.253164:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066648192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768323.253171:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768323.253173:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768323.253175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768323.253179:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5154 00000100:00080000:3.0:1775768323.253181:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768323.253184:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768323.253187:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768323.253199:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066648320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768323.253208:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768323.253210:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768323.253212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768323.253216:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5154 00000100:00080000:3.0:1775768323.253219:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768323.253223:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768323.253227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768323.253228:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775768323.253240:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066648448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768323.253275:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768323.253295:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768323.253313:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768323.893100:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768324.277113:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768324.917103:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768325.301060:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768325.941138:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768326.325289:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:0.0:1775768326.325310:0:10933:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:0.0:1775768326.325364:0:10933:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1775768326.325410:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775768326.325468:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775768326.325479:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775768326.325494:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:0.0:1775768326.325502:0:10933:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775768326.325706:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768323/real 1775768326] req@ffff9d8188742300 x1862023066648448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768326.325736:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066648448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768326.325754:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768326.325760:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768326.325764:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648448:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768326.325773:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768323/real 1775768326] req@ffff9d8188743800 x1862023066648064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768326.325786:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066648064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768326.325794:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768326.325798:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768326.325802:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768326.325809:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768323/real 1775768326] req@ffff9d8188741880 x1862023066648192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768326.325821:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066648192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768326.325829:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768326.325833:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768326.325836:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768326.325843:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768323/real 1775768326] req@ffff9d8188742680 x1862023066648320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768326.325854:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066648320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768326.325862:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768326.325866:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768326.325907:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768326.325916:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768323/real 1775768326] req@ffff9d8188741f80 x1862023066647936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768326.325930:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066647936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768326.325938:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768326.325942:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768326.325945:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066647936:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775768326.965119:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768327.349111:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768327.989025:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768328.373043:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768328.373054:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768328.373060:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768328.373071:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5159 00000100:00100000:2.0:1775768328.373081:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768328.373084:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768328.373093:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768328.373137:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066648576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768328.373165:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768328.373177:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768328.373179:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768328.373183:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5159 00000100:00080000:0.0:1775768328.373189:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768328.373194:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768328.373205:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768328.373219:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066648704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768328.373222:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066648576:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768328.373237:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768328.373241:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768328.373243:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768328.373247:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5159 00000100:00080000:0.0:1775768328.373251:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768328.373254:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768328.373258:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768328.373269:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066648832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768328.373279:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768328.373283:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768328.373285:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768328.373288:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5159 00000100:00080000:0.0:1775768328.373302:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768328.373306:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768328.373311:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775768328.373319:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768328.373322:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066648960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768328.373331:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768328.373335:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768328.373337:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768328.373342:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5159 00000100:00080000:0.0:1775768328.373345:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768328.373348:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768328.373353:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768328.373362:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066649088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768328.373379:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768328/real 1775768328] req@ffff9d81901ad180 x1862023066648576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768328.373401:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066648576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768328.373412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768328.373417:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768328.373421:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066648576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768328.373450:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648704:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768328.373487:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768328.373499:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768328/real 1775768328] req@ffff9d81901aed80 x1862023066648704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768328.373510:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066648704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768328.373518:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768328.373521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768328.373523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768328.373528:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648832:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768328.373547:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768328.373557:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768328/real 1775768328] req@ffff9d81901af100 x1862023066648832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768328.373568:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066648832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768328.373582:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768328.373585:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768328.373588:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768328.373594:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648960:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768328.373617:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768328.373626:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768328/real 1775768328] req@ffff9d81901ae680 x1862023066648960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768328.373637:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066648960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768328.373645:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768328.373649:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768328.373652:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066648960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768328.373658:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649088:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768328.373675:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768328.373684:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768328/real 1775768328] req@ffff9d81901afb80 x1862023066649088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768328.373695:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066649088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768328.373705:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768328.373707:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768328.373711:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768329.013111:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768329.397151:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768330.039603:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768330.420969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768331.061083:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768331.445006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768332.085067:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768332.469064:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768333.109062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768333.493002:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768333.493008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768333.493013:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768333.493017:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768333.493026:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5165 00000100:00080000:1.0:1775768333.493036:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768333.493044:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768333.493079:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066649216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768333.493109:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768333.493112:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768333.493114:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768333.493118:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5165 00000100:00080000:1.0:1775768333.493122:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768333.493126:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768333.493135:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768333.493142:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066649216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768333.493149:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066649344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768333.493160:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768333.493163:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768333.493165:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768333.493169:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5165 00000100:00080000:1.0:1775768333.493188:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768333.493192:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768333.493195:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1775768333.493198:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768333.493207:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066649472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768333.493213:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768333.493215:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768333.493217:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768333.493219:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5165 00000100:00080000:1.0:1775768333.493222:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768333.493224:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768333.493227:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768333.493234:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066649600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768333.493239:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768333.493242:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768333.493243:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768333.493245:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5165 00000100:00080000:1.0:1775768333.493248:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768333.493250:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768333.493255:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768333.493257:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768333/real 1775768333] req@ffff9d8190551880 x1862023066649216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768333.493268:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066649728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768333.493278:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066649216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768333.493299:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768333.493304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768333.493308:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066649216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768333.493330:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768333.493352:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768333.493365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768333/real 1775768333] req@ffff9d8190551500 x1862023066649344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768333.493376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066649344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768333.493384:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768333.493387:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768333.493390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768333.493403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768333.493417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768333.493425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768333/real 1775768333] req@ffff9d8190553100 x1862023066649472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768333.493432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066649472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768333.493437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768333.493439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768333.493440:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768333.493444:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768333.493461:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768333.493469:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768333/real 1775768333] req@ffff9d8190550e00 x1862023066649600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768333.493480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066649600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768333.493488:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768333.493492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768333.493494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768333.493500:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768333.493517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768333.493526:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768333/real 1775768333] req@ffff9d8190553480 x1862023066649728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768333.493537:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066649728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768333.493545:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768333.493548:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768333.493551:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768334.133045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768334.517004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768335.157049:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768335.540966:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768336.181074:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768336.565033:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768337.205031:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768337.588986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768338.229107:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768338.613002:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768338.613008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768338.613011:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768338.613017:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768338.613026:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5170 00000100:00080000:1.0:1775768338.613035:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768338.613042:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768338.613077:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066649856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768338.613105:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768338.613109:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768338.613111:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768338.613115:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5170 00000100:00080000:1.0:1775768338.613118:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768338.613121:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768338.613129:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768338.613140:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066649984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768338.613149:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768338.613152:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768338.613154:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768338.613159:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066649856:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768338.613161:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5170 00000100:00080000:1.0:1775768338.613168:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768338.613172:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768338.613176:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768338.613187:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066650112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768338.613197:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768338.613200:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768338.613201:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768338.613205:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5170 00000100:00080000:1.0:1775768338.613211:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768338.613214:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768338.613219:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768338.613222:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768338.613231:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066650240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768338.613241:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768338.613244:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768338.613246:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768338.613249:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5170 00000100:00080000:1.0:1775768338.613253:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768338.613257:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768338.613262:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768338.613269:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768338/real 1775768338] req@ffff9d8190553480 x1862023066649856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768338.613274:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066650368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768338.613289:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066649856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768338.613297:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768338.613301:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768338.613304:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066649856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768338.613309:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768338.613325:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768338.613333:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768338/real 1775768338] req@ffff9d8190550e00 x1862023066649984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768338.613343:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066649984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768338.613355:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768338.613358:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768338.613361:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066649984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768338.613382:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768338.613401:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768338.613412:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768338/real 1775768338] req@ffff9d8190553100 x1862023066650112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768338.613423:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066650112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768338.613429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768338.613432:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768338.613435:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768338.613440:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768338.613458:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768338.613467:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768338/real 1775768338] req@ffff9d8190551500 x1862023066650240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768338.613478:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066650240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768338.613487:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768338.613490:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768338.613494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768338.613499:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768338.613515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768338.613524:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768338/real 1775768338] req@ffff9d8190551880 x1862023066650368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768338.613535:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066650368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768338.613544:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768338.613547:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768338.613549:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650368:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768339.253062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768339.636989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768340.277049:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768340.660943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768341.301037:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768341.684997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768342.325051:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768342.708958:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768343.349040:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768343.732984:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768343.732989:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768343.732992:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768343.732998:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768343.733007:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5175 00000100:00080000:1.0:1775768343.733016:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768343.733022:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768343.733060:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066650496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768343.733092:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768343.733096:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768343.733098:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768343.733101:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5175 00000100:00080000:1.0:1775768343.733105:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768343.733108:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768343.733117:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768343.733129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066650624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768343.733138:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768343.733140:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768343.733142:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768343.733146:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5175 00000100:00080000:1.0:1775768343.733148:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768343.733151:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768343.733155:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768343.733164:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066650752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768343.733184:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768343.733188:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768343.733189:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768343.733193:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5175 00000100:00080000:1.0:1775768343.733196:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768343.733199:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768343.733203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768343.733212:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066650880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768343.733220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768343.733222:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768343.733224:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768343.733227:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5175 00000100:00080000:1.0:1775768343.733229:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768343.733232:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768343.733235:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768343.733244:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066651008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768343.733332:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066650496:192.168.202.105@tcp:250:kworker.0 00000800:00000100:0.0:1775768343.733386:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768343.733428:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768343/real 1775768343] req@ffff9d8190551880 x1862023066650496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768343.733446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066650496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768343.733454:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768343.733457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768343.733461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066650496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768343.733468:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768343.733489:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768343.733500:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768343/real 1775768343] req@ffff9d8190551500 x1862023066650624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768343.733512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066650624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768343.733519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768343.733522:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768343.733525:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768343.733530:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768343.733549:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768343.733560:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768343/real 1775768343] req@ffff9d8190553100 x1862023066650752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768343.733571:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066650752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768343.733578:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768343.733581:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768343.733584:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768343.733589:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768343.733608:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768343.733618:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768343/real 1775768343] req@ffff9d8190550e00 x1862023066650880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768343.733630:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066650880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768343.733639:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768343.733642:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768343.733645:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066650880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768343.733650:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768343.733669:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768343.733678:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768343/real 1775768343] req@ffff9d8190553480 x1862023066651008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768343.733689:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066651008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768343.733697:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768343.733700:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768343.733702:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768344.373051:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768344.757681:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768345.397121:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768345.781004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768346.421328:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768346.805034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768347.445043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768347.829052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768348.469110:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768348.853020:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1775768348.853022:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768348.853032:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768348.853040:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768348.853058:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5180 00000100:00080000:3.0:1775768348.853070:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768348.853076:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768348.853118:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066651136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768348.853152:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768348.853156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768348.853158:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768348.853162:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5180 00000100:00080000:3.0:1775768348.853166:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768348.853170:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768348.853183:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768348.853194:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066651264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768348.853200:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768348.853204:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768348.853205:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768348.853208:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5180 00000100:00080000:3.0:1775768348.853210:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768348.853214:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768348.853217:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768348.853226:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066651392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768348.853234:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768348.853239:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768348.853240:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768348.853244:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066651136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768348.853245:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5180 00000100:00080000:3.0:1775768348.853252:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768348.853261:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768348.853266:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768348.853289:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066651520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768348.853299:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768348.853303:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768348.853307:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768348.853310:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5180 00000100:00080000:3.0:1775768348.853315:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768348.853320:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768348.853323:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768348.853330:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768348.853341:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066651648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768348.853387:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768348/real 1775768348] req@ffff9d8188740e00 x1862023066651136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768348.853405:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066651136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768348.853415:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768348.853419:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768348.853422:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066651136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768348.853432:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768348.853454:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768348.853473:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768348/real 1775768348] req@ffff9d8188741180 x1862023066651264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768348.853482:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066651264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768348.853487:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768348.853489:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768348.853491:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768348.853510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768348.853529:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768348.853538:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768348/real 1775768348] req@ffff9d8188743100 x1862023066651392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768348.853549:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066651392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768348.853556:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768348.853560:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768348.853563:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768348.853568:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768348.853586:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768348.853596:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768348/real 1775768348] req@ffff9d8188742d80 x1862023066651520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768348.853608:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066651520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768348.853617:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768348.853620:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768348.853623:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768348.853627:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768348.853641:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768348.853647:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768348/real 1775768348] req@ffff9d8188743b80 x1862023066651648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768348.853654:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066651648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768348.853659:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768348.853661:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768348.853663:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768349.493162:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768349.877021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768350.517037:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768350.901044:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768351.541180:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768351.925368:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768352.565105:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768352.949100:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768353.589048:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768353.972982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768353.972986:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768353.972991:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768353.972995:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768353.973002:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5185 00000100:00080000:1.0:1775768353.973012:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768353.973019:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768353.973055:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066651776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768353.973091:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768353.973095:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768353.973100:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768353.973104:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5185 00000100:00080000:1.0:1775768353.973114:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768353.973121:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768353.973128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768353.973136:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066651776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768353.973141:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066651904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768353.973149:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768353.973151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768353.973153:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768353.973156:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5185 00000100:00080000:1.0:1775768353.973158:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768353.973160:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768353.973163:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768353.973171:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066652032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768353.973177:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768353.973180:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768353.973181:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768353.973188:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5185 00000800:00000100:0.0:1775768353.973196:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768353.973202:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768353.973208:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768353.973214:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768353.973227:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066652160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768353.973236:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768353.973239:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768353.973240:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768353.973244:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5185 00000100:00080000:1.0:1775768353.973248:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768353.973252:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768353.973253:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768353/real 1775768353] req@ffff9d8190553480 x1862023066651776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1775768353.973258:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768353.973274:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066652288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768353.973277:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066651776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768353.973300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768353.973304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768353.973307:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066651776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768353.973328:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768353.973349:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768353.973359:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768353/real 1775768353] req@ffff9d8190550e00 x1862023066651904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768353.973528:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066651904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768353.973534:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768353.973536:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768353.973538:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066651904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768353.973543:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768353.973557:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768353.973564:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768353/real 1775768353] req@ffff9d8190553100 x1862023066652032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768353.973571:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066652032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768353.973576:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768353.973579:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768353.973582:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768353.973587:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768353.973604:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768353.973613:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768353/real 1775768353] req@ffff9d8190551500 x1862023066652160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768353.973624:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066652160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768353.973632:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768353.973635:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768353.973638:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768353.973643:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768353.973660:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768353.973671:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768353/real 1775768353] req@ffff9d8190551880 x1862023066652288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768353.973682:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066652288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768353.973690:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768353.973693:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768353.973696:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768354.614002:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768354.998115:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768355.637081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768356.020986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768356.661063:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768357.046370:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768357.685174:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768358.069043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768358.709035:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1775768359.092996:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768359.093006:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768359.093010:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768359.093032:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5190 00000100:00080000:1.0:1775768359.093042:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768359.093049:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768359.093087:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066652416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768359.093173:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066652416:192.168.202.105@tcp:250:kworker.0 00000800:00000100:0.0:1775768359.093230:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:1.0:1775768359.093264:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768359.093270:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768359.093272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768359.093277:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5190 00000100:00080000:1.0:1775768359.093282:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768359.093286:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768359.093295:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768359.093310:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066652544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1775768359.093404:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:0.0:1775768359.093568:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768359/real 1775768359] req@ffff9d8190551880 x1862023066652416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768359.093587:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066652416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768359.093596:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768359.093600:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768359.093604:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066652416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768359.093626:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768359.093650:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768359.093662:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768359/real 1775768359] req@ffff9d8190551500 x1862023066652544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768359.093674:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066652544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768359.093681:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768359.093683:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768359.093686:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652544:192.168.202.105@tcp:38:kworker.0 00000100:00080000:1.0:1775768359.094308:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768359.094313:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768359.094315:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768359.094320:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5190 00000100:00080000:1.0:1775768359.094325:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768359.094330:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768359.094336:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768359.094353:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066652672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768359.094364:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768359.094368:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768359.094369:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768359.094373:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5190 00000100:00080000:1.0:1775768359.094375:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768359.094379:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768359.094384:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768359.094393:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066652800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768359.094401:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768359.094404:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768359.094406:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768359.094408:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5190 00000100:00080000:1.0:1775768359.094412:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768359.094415:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768359.094419:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768359.094426:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066652928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768359.094489:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768359.094514:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768359.094528:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768359/real 1775768359] req@ffff9d8190553100 x1862023066652672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768359.094540:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066652672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768359.094554:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768359.094557:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768359.094560:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768359.094566:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768359.094584:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768359.094593:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768359/real 1775768359] req@ffff9d8190550e00 x1862023066652800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768359.094604:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066652800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768359.094612:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768359.094615:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768359.094618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768359.094624:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768359.094641:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768359.094651:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768359/real 1775768359] req@ffff9d8190553480 x1862023066652928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768359.094661:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066652928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768359.094668:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768359.094671:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768359.094674:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066652928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768359.733168:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768360.117001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768360.757043:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768361.140994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768361.781086:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768362.165001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768362.805038:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768363.189005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768363.829063:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768364.212987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768364.213004:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768364.213009:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768364.213013:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768364.213022:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5195 00000100:00080000:3.0:1775768364.213032:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768364.213040:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768364.213075:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066653056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768364.213094:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768364.213096:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768364.213097:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768364.213100:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5195 00000100:00080000:3.0:1775768364.213102:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768364.213105:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768364.213112:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768364.213119:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066653184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768364.213129:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768364.213133:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066653056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768364.213136:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768364.213139:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768364.213145:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5195 00000100:00080000:3.0:1775768364.213148:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768364.213152:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768364.213157:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768364.213169:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066653312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768364.213177:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768364.213181:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768364.213187:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768364.213191:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5195 00000100:00080000:3.0:1775768364.213194:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1775768364.213198:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768364.213202:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768364.213208:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768364.213221:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066653440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768364.213230:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768364.213234:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768364.213235:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768364.213240:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5195 00000100:00080000:3.0:1775768364.213243:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768364.213248:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768364/real 1775768364] req@ffff9d8188743480 x1862023066653056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768364.213250:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768364.213257:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768364.213273:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066653568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768364.213274:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066653056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768364.213300:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768364.213306:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768364.213310:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066653056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768364.213331:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768364.213353:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768364.213363:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768364/real 1775768364] req@ffff9d8188740a80 x1862023066653184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768364.213374:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066653184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768364.213382:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768364.213385:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768364.213388:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768364.213393:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768364.213411:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768364.213421:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768364/real 1775768364] req@ffff9d8188741500 x1862023066653312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768364.213432:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066653312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768364.213439:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768364.213442:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768364.213445:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768364.213450:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768364.213468:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768364.213478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768364/real 1775768364] req@ffff9d8188743b80 x1862023066653440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768364.213489:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066653440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768364.213499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768364.213502:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768364.213505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768364.213510:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768364.214143:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768364.214157:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768364/real 1775768364] req@ffff9d8188742d80 x1862023066653568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768364.214170:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066653568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768364.214179:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768364.214182:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768364.214185:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768364.853097:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768365.237063:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768365.877897:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768366.260998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768366.901082:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768367.284990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768367.925036:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768368.308999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768368.949038:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768369.333008:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768369.333017:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768369.333022:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768369.333031:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5200 00000100:00080000:2.0:1775768369.333041:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768369.333048:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768369.333085:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0a80 x1862023066653696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768369.333115:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768369.333118:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768369.333120:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768369.333123:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5200 00000100:00080000:2.0:1775768369.333126:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768369.333130:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768369.333139:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768369.333152:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066653824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768369.333162:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768369.333165:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768369.333167:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768369.333173:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066653696:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768369.333173:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5200 00000100:00080000:2.0:1775768369.333178:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768369.333183:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768369.333188:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768369.333200:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066653952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768369.333210:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768369.333213:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768369.333215:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768369.333219:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5200 00000100:00080000:2.0:1775768369.333225:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768369.333229:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768369.333234:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768369.333237:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775768369.333247:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066654080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768369.333256:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768369.333260:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768369.333261:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768369.333266:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5200 00000100:00080000:2.0:1775768369.333269:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768369.333273:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768369.333277:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775768369.333288:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066654208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768369.333300:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768369/real 1775768369] req@ffff9d81984a0a80 x1862023066653696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768369.333322:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00100000:0.0:1775768369.333323:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0a80 x1862023066653696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768369.333335:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768369.333340:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768369.333344:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066653696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768369.333351:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768369.333374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768369.333385:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768369/real 1775768369] req@ffff9d81984a1500 x1862023066653824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768369.333397:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066653824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768369.333404:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768369.333406:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768369.333409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768369.333428:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768369.333444:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768369.333452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768369/real 1775768369] req@ffff9d81984a3100 x1862023066653952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768369.333462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066653952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768369.333469:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768369.333471:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768369.333474:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066653952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768369.333479:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768369.333495:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768369.333503:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768369/real 1775768369] req@ffff9d81984a3800 x1862023066654080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768369.333513:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066654080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768369.333522:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768369.333525:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768369.333528:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768369.333532:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768369.333547:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768369.333555:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768369/real 1775768369] req@ffff9d81984a1880 x1862023066654208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768369.333565:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066654208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768369.333589:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768369.333592:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768369.333595:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768369.973105:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768370.357014:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768370.997074:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768371.381009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768372.021066:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768372.406438:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768373.045062:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768373.428944:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768374.069067:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768374.453010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768374.453010:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768374.453021:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768374.453025:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768374.453035:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5205 00000100:00080000:3.0:1775768374.453046:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768374.453054:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768374.453087:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066654336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768374.453110:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768374.453112:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768374.453113:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768374.453116:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5205 00000100:00080000:3.0:1775768374.453118:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768374.453121:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768374.453127:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768374.453135:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066654464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768374.453141:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768374.453143:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768374.453144:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768374.453147:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5205 00000100:00080000:3.0:1775768374.453149:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768374.453151:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768374.453153:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768374.453156:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066654336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768374.453162:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066654592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768374.453170:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768374.453174:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768374.453175:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768374.453179:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5205 00000100:00080000:3.0:1775768374.453182:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768374.453185:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768374.453189:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768374.453210:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066654720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768374.453220:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1775768374.453225:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768374.453225:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768374.453229:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768374.453234:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5205 00000100:00080000:3.0:1775768374.453239:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768374.453243:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768374.453248:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768374.453261:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066654848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768374.453274:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768374/real 1775768374] req@ffff9d8188743100 x1862023066654336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768374.453293:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066654336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768374.453301:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768374.453304:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768374.453307:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066654336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768374.453313:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768374.453336:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768374.453344:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768374/real 1775768374] req@ffff9d8188741180 x1862023066654464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768374.453352:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066654464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768374.453360:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768374.453362:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768374.453364:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768374.453380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768374.453390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768374.453396:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768374/real 1775768374] req@ffff9d8188740e00 x1862023066654592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768374.453402:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066654592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768374.453407:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768374.453408:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768374.453410:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768374.453413:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768374.453422:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768374.453427:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768374/real 1775768374] req@ffff9d8188741f80 x1862023066654720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768374.453433:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066654720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768374.453438:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768374.453440:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768374.453442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768374.453453:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768374.453471:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768374.453480:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768374/real 1775768374] req@ffff9d8188742680 x1862023066654848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768374.453491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066654848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768374.453499:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768374.453502:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768374.453505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066654848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768375.093043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768375.476953:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768376.117110:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768376.500985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768377.141084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768377.524963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768378.165085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768378.548990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768379.189092:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768379.573012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768379.576730:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768379.576741:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768379.576747:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768379.576760:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5211 00000100:00080000:3.0:1775768379.576774:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768379.576782:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768379.576829:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066654976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768379.576863:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768379.576867:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768379.576869:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768379.576895:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5211 00000100:00080000:3.0:1775768379.576900:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768379.576905:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768379.576916:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768379.576919:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066654976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768379.576932:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066655104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768379.576943:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768379.576947:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768379.576949:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768379.576953:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5211 00000100:00080000:3.0:1775768379.576958:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768379.576962:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768379.576968:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768379.576980:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066655232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768379.576989:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768379.576993:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768379.576995:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1775768379.576999:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768379.577001:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5211 00000100:00080000:3.0:1775768379.577006:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768379.577010:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768379.577016:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768379.577030:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066655360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768379.577039:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768379.577042:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768379.577044:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768379.577048:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5211 00000100:00080000:3.0:1775768379.577052:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768379.577056:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768379.577061:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768379.577073:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768379/real 1775768379] req@ffff9d8188741880 x1862023066654976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768379.577073:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066655488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768379.577112:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066654976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768379.577125:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768379.577130:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768379.577134:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066654976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768379.577161:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768379.577185:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768379.577199:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768379/real 1775768379] req@ffff9d8188743800 x1862023066655104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768379.577213:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066655104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768379.577221:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768379.577224:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768379.577227:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768379.577233:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768379.577253:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768379.577265:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768379/real 1775768379] req@ffff9d8188742300 x1862023066655232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768379.577276:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066655232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768379.577298:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768379.577301:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768379.577304:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768379.577310:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768379.577332:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768379.577342:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768379/real 1775768379] req@ffff9d8188742a00 x1862023066655360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768379.577354:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066655360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768379.577364:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768379.577376:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768379.577380:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768379.577385:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768379.577404:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768379.577414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768379/real 1775768379] req@ffff9d8188740700 x1862023066655488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768379.577426:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066655488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768379.577435:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768379.577439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768379.577442:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768380.214503:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768380.596995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768381.237340:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768381.621009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768382.261113:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768382.645016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768383.285753:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768383.669477:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768384.310826:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768384.692954:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768384.693010:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768384.693016:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768384.693020:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768384.693028:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5216 00000100:00080000:0.0:1775768384.693037:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768384.693043:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768384.693078:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066655616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768384.693105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768384.693109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768384.693111:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768384.693114:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5216 00000100:00080000:0.0:1775768384.693117:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768384.693120:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768384.693128:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768384.693141:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066655744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768384.693151:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768384.693154:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066655616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768384.693156:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768384.693159:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768384.693166:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5216 00000100:00080000:0.0:1775768384.693169:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768384.693173:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768384.693178:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768384.693192:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066655872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768384.693201:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768384.693205:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768384.693206:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768384.693209:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5216 00000100:00080000:0.0:1775768384.693213:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768384.693216:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775768384.693220:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768384.693225:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768384.693236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066656000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768384.693245:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768384.693248:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768384.693250:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768384.693253:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5216 00000100:00080000:0.0:1775768384.693257:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768384.693261:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768384/real 1775768384] req@ffff9d81901afb80 x1862023066655616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768384.693262:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768384.693270:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768384.693284:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066656128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768384.693287:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066655616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768384.693310:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768384.693315:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768384.693319:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066655616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768384.693352:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768384.693374:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768384.693386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768384/real 1775768384] req@ffff9d81901ae680 x1862023066655744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768384.693398:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066655744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768384.693406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768384.693409:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768384.693412:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768384.693418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768384.693433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768384.693443:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768384/real 1775768384] req@ffff9d81901af100 x1862023066655872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768384.693474:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066655872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768384.693482:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768384.693485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768384.693489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066655872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768384.693494:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768384.693511:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768384.693519:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768384/real 1775768384] req@ffff9d81901aed80 x1862023066656000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768384.693534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066656000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768384.693541:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768384.693544:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768384.693546:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768384.693550:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768384.693563:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768384.693570:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768384/real 1775768384] req@ffff9d81901ad180 x1862023066656128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768384.693579:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066656128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768384.693585:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768384.693587:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768384.693590:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768385.333504:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768385.717016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768386.357227:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768386.741108:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768387.381020:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768387.764999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768388.405084:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768388.788985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768389.429087:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768389.813112:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768389.813122:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768389.813127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768389.813137:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5221 00000100:00080000:0.0:1775768389.813148:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768389.813155:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768389.813209:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066656256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768389.813239:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768389.813242:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768389.813244:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768389.813248:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5221 00000100:00080000:0.0:1775768389.813252:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768389.813256:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768389.813265:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768389.813279:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066656384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768389.813287:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066656256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768389.815381:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768389.816512:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768389.816518:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768389.816521:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768389.816528:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5221 00000100:00080000:0.0:1775768389.816535:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768389.816540:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768389.816550:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768389.816578:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066656512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768389.816588:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768389.816591:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768389.816593:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768389.816596:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5221 00000100:00080000:0.0:1775768389.816600:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768389.816603:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768389.816608:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768389.816617:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066656640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768389.816631:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768389.816634:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768389.816635:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768389.816639:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5221 00000100:00080000:0.0:1775768389.816642:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768389.816644:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768389.816648:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768389.816657:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066656768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768389.817773:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768389.817844:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768389.817860:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768389.817873:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768390.453085:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768390.839521:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768391.477045:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768391.860986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768392.501076:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768392.884993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:3.0:1775768392.885128:0:10932:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:3.0:1775768392.885162:0:10932:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1775768392.885177:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775768392.885216:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775768392.885225:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775768392.885232:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:3.0:1775768392.885239:0:10932:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775768392.885347:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768389/real 1775768392] req@ffff9d81901aed80 x1862023066656384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768392.885370:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066656384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768392.885399:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768392.885404:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768392.885408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768392.885415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768389/real 1775768392] req@ffff9d81901af100 x1862023066656512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768392.885429:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066656512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768392.885437:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768392.885441:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768392.885444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768392.885450:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768389/real 1775768392] req@ffff9d81901ae680 x1862023066656640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768392.885462:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066656640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768392.885471:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768392.885474:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768392.885478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768392.885484:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768389/real 1775768392] req@ffff9d81901afb80 x1862023066656768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768392.885494:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066656768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768392.885502:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768392.885510:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768392.885514:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066656768:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768392.885520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768389/real 1775768392] req@ffff9d81901ad180 x1862023066656256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768392.885533:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066656256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768392.885541:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768392.885544:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768392.885547:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066656256:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775768393.525071:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768393.909451:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768394.549055:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768394.932959:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768394.932970:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1775768394.932971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768394.932973:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768394.932984:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5226 00000100:00080000:0.0:1775768394.932995:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768394.933002:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768394.933034:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066656896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768394.933061:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768394.933064:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768394.933066:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768394.933080:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5226 00000100:00080000:0.0:1775768394.933084:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768394.933088:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768394.933095:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066656896:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775768394.933099:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768394.933112:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066657024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768394.933122:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768394.933126:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768394.933127:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768394.933132:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5226 00000100:00080000:0.0:1775768394.933146:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768394.933150:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:1.0:1775768394.933155:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:0.0:1775768394.933156:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768394.933169:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066657152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768394.933179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768394.933182:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768394.933184:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768394.933188:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5226 00000100:00080000:0.0:1775768394.933192:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768394.933196:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768394.933201:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768394/real 1775768394] req@ffff9d81901ad180 x1862023066656896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768394.933202:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768394.933216:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066657280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768394.933225:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768394.933230:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066656896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768394.933231:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768394.933247:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768394.933251:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768394.933255:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066656896:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768394.933275:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657024:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768394.933297:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768394.933309:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768394/real 1775768394] req@ffff9d81901afb80 x1862023066657024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768394.933320:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066657024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768394.933326:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768394.933329:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768394.933332:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768394.933337:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657152:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768394.933372:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768394.933382:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768394/real 1775768394] req@ffff9d81901ae680 x1862023066657152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768394.933391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066657152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768394.933399:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768394.933402:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768394.933406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768394.933411:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657280:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768394.933429:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768394.933438:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768394/real 1775768394] req@ffff9d81901af100 x1862023066657280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768394.933449:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066657280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768394.933457:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768394.933459:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768394.933462:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657280:192.168.202.105@tcp:8:kworker.0 00000100:00080000:0.0:1775768394.933678:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768394.933683:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5226 00000100:00080000:0.0:1775768394.933687:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768394.933691:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768394.933697:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768394.933709:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066657408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768394.934177:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657408:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768394.934205:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768394.934219:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768394/real 1775768394] req@ffff9d81901aed80 x1862023066657408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768394.934230:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066657408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768394.934239:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768394.934242:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768394.934245:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768395.573071:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768395.957471:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768396.597133:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768396.981019:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768397.621060:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768398.004982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768398.645051:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768399.028988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768399.669139:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1775768400.052988:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768400.053008:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768400.053015:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768400.053019:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768400.053028:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5231 00000100:00080000:3.0:1775768400.053038:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768400.053045:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768400.053081:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066657536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768400.053105:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768400.053109:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768400.053110:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768400.053115:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5231 00000100:00080000:3.0:1775768400.053119:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768400.053123:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768400.053132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066657536:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768400.053133:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768400.053151:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066657664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768400.053162:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768400.053165:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768400.053167:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768400.053179:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5231 00000100:00080000:3.0:1775768400.053183:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768400.053187:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768400.053189:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768400.053196:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768400.053209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066657792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768400.053218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768400.053222:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768400.053224:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768400.053228:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5231 00000100:00080000:3.0:1775768400.053232:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768400.053235:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768400/real 1775768400] req@ffff9d8188740000 x1862023066657536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768400.053239:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768400.053247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768400.053260:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066657536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768400.053262:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066657920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768400.053275:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1775768400.053279:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768400.053281:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:3.0:1775768400.053286:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1775768400.053287:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066657536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768400.053289:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768400.053297:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5231 00000100:00080000:3.0:1775768400.053301:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768400.053317:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768400.053323:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768400.053327:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:3.0:1775768400.053338:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066658048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768400.053352:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768400.053365:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768400/real 1775768400] req@ffff9d8188741c00 x1862023066657664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768400.053378:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066657664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768400.053386:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768400.053388:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768400.053390:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768400.053394:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768400.053407:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768400.053414:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768400/real 1775768400] req@ffff9d8188740380 x1862023066657792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768400.053421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066657792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768400.053426:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768400.053428:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768400.053430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768400.053433:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768400.053446:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768400.053454:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768400/real 1775768400] req@ffff9d8188740700 x1862023066657920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768400.053465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066657920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768400.053474:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768400.053476:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768400.053478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066657920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768400.053488:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768400.053501:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768400.053507:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768400/real 1775768400] req@ffff9d8188742a00 x1862023066658048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768400.053514:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066658048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768400.053519:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768400.053521:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768400.053523:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768400.693171:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768401.076959:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768401.717083:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768402.101005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768402.741039:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768403.125027:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768403.765068:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768404.149047:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768404.789317:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768405.172990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768405.172991:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768405.172999:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768405.173003:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768405.173010:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5236 00000100:00080000:0.0:1775768405.173018:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768405.173023:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768405.173054:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066658176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768405.173072:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768405.173074:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768405.173075:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768405.173077:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5236 00000100:00080000:0.0:1775768405.173079:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768405.173081:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768405.173088:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768405.173095:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066658304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768405.173103:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768405.173104:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066658176:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768405.173105:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768405.173108:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768405.173115:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5236 00000100:00080000:0.0:1775768405.173118:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768405.173122:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768405.173126:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768405.173136:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066658432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768405.173145:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768405.173149:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768405.173150:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768405.173154:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5236 00000100:00080000:0.0:1775768405.173165:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768405.173170:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768405.173174:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775768405.173174:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768405.173188:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066658560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768405.173197:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768405.173200:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768405.173202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768405.173205:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5236 00000100:00080000:0.0:1775768405.173209:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768405.173213:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768405.173218:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768405.173230:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066658688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768405.173233:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768405/real 1775768405] req@ffff9d81901aed80 x1862023066658176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768405.173268:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066658176/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768405.173276:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768405.173278:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768405.173281:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066658176:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768405.173298:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658304:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768405.173318:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768405.173329:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768405/real 1775768405] req@ffff9d81901af100 x1862023066658304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768405.173341:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066658304/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768405.173350:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768405.173353:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768405.173356:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658304:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768405.173362:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658432:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768405.173378:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768405.173385:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768405/real 1775768405] req@ffff9d81901ae680 x1862023066658432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768405.173392:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066658432/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768405.173397:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768405.173399:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768405.173400:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658432:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768405.173403:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658560:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768405.173417:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768405.173426:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768405/real 1775768405] req@ffff9d81901afb80 x1862023066658560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768405.173436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066658560/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768405.173445:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768405.173448:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768405.173451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658560:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768405.173457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658688:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768405.173474:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768405.173485:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768405/real 1775768405] req@ffff9d81901ad180 x1862023066658688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768405.173496:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066658688/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768405.173504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768405.173507:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768405.173509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658688:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768405.813056:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768406.196985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768406.837052:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768407.221366:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768407.861060:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768408.244980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768408.885016:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768409.268984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768409.909058:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1775768410.292976:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768410.292976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768410.292985:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768410.292989:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768410.292997:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5241 00000100:00080000:0.0:1775768410.293007:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768410.293014:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768410.293050:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066658816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768410.293087:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768410.293090:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768410.293092:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768410.293096:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5241 00000100:00080000:0.0:1775768410.293100:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768410.293104:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768410.293113:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768410.293124:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066658944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768410.293133:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768410.293138:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768410.293140:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768410.293143:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5241 00000100:00080000:0.0:1775768410.293146:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768410.293149:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768410.293154:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768410.293162:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066659072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768410.293172:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768410.293175:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768410.293177:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768410.293181:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5241 00000100:00080000:0.0:1775768410.293185:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768410.293189:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768410.293193:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066658816:192.168.202.105@tcp:250:kworker.0 00000008:00100000:0.0:1775768410.293195:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768410.293218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066659200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768410.293229:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768410.293233:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768410.293245:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768410.293250:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5241 00000100:00080000:0.0:1775768410.293254:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768410.293258:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775768410.293261:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768410.293267:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768410.293278:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066659328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768410.293303:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768410/real 1775768410] req@ffff9d81901ad180 x1862023066658816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768410.293320:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066658816/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768410.293330:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768410.293334:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768410.293339:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066658816:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768410.293348:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658944:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768410.293369:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768410.293380:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768410/real 1775768410] req@ffff9d81901afb80 x1862023066658944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768410.293391:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066658944/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768410.293400:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768410.293403:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768410.293406:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066658944:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768410.293411:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659072:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768410.293436:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768410.293445:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768410/real 1775768410] req@ffff9d81901ae680 x1862023066659072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768410.293456:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066659072/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768410.293464:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768410.293467:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768410.293470:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659072:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768410.293490:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659200:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768410.293504:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768410.293512:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768410/real 1775768410] req@ffff9d81901af100 x1862023066659200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768410.293524:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066659200/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768410.293533:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768410.293536:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768410.293539:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659200:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768410.293543:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659328:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768410.293555:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768410.293563:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768410/real 1775768410] req@ffff9d81901aed80 x1862023066659328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768410.293574:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066659328/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768410.293581:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768410.293584:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768410.293586:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659328:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768410.933043:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768411.317006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768411.957066:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768412.341178:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768412.981161:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768413.364980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768414.005084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768414.389043:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768415.029249:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768415.412963:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768415.413027:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768415.413036:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768415.413042:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768415.413054:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5246 00000100:00080000:3.0:1775768415.413067:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768415.413075:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768415.413153:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066659456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768415.413197:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768415.413201:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768415.413208:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768415.413211:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5246 00000100:00080000:3.0:1775768415.413215:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768415.413217:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768415.413227:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768415.413236:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066659584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768415.413243:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768415.413245:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768415.413246:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768415.413250:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5246 00000100:00080000:3.0:1775768415.413252:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768415.413254:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768415.413257:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768415.413263:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066659712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768415.413268:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768415.413271:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768415.413272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768415.413276:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5246 00000100:00080000:3.0:1775768415.413279:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768415.413282:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768415.413286:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768415.413298:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066659840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768415.413307:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768415.413310:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768415.413312:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768415.413315:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5246 00000100:00080000:3.0:1775768415.413319:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768415.413322:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768415.413327:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768415.413337:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066659968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768415.413621:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066659456:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775768415.413680:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768415.413740:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768415/real 1775768415] req@ffff9d8188742300 x1862023066659456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768415.413758:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066659456/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768415.413768:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768415.413773:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768415.413777:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066659456:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768415.413785:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659584:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768415.413807:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768415.413819:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768415/real 1775768415] req@ffff9d8188743800 x1862023066659584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768415.413827:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066659584/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768415.413832:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768415.413834:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768415.413836:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659584:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768415.413840:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659712:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768415.413857:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768415.413866:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768415/real 1775768415] req@ffff9d8188741880 x1862023066659712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768415.413903:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066659712/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768415.413916:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768415.413919:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768415.413922:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659712:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768415.413927:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659840:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768415.413948:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768415.413958:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768415/real 1775768415] req@ffff9d8188742680 x1862023066659840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768415.413968:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066659840/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768415.413974:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768415.413977:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768415.413980:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659840:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768415.413985:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659968:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768415.414000:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768415.414006:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768415/real 1775768415] req@ffff9d8188741f80 x1862023066659968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768415.414017:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066659968/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768415.414025:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768415.414028:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768415.414032:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066659968:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768416.053073:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768416.436986:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768417.077115:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768417.460984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768418.101068:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768418.484996:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768419.125070:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768419.509000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768420.149061:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768420.532994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768420.533176:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768420.533182:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768420.533186:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768420.533196:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5252 00000100:00080000:0.0:1775768420.533206:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768420.533213:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768420.533250:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066660096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768420.533273:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768420.533277:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768420.533279:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768420.533283:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5252 00000100:00080000:0.0:1775768420.533287:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768420.533292:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066660096:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768420.533293:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768420.533305:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768420.533320:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066660224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768420.533330:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768420.533333:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768420.533334:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768420.533338:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5252 00000100:00080000:0.0:1775768420.533348:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768420.533352:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768420.533357:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768420.533364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768420.533367:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066660352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768420.533374:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768420.533376:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768420.533378:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768420.533381:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5252 00000100:00080000:0.0:1775768420.533386:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768420.533388:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768420.533391:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768420.533398:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066660480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768420.533403:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768420.533406:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768420.533407:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768420.533409:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5252 00000100:00080000:0.0:1775768420.533411:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768420.533415:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768420.533415:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768420/real 1775768420] req@ffff9d81901aed80 x1862023066660096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1775768420.533417:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768420.533427:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066660608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768420.533439:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066660096/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768420.533453:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768420.533457:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768420.533461:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066660096:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768420.533482:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660224:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768420.533501:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768420.533511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768420/real 1775768420] req@ffff9d81901af100 x1862023066660224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768420.533522:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066660224/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768420.533541:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768420.533545:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768420.533548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660224:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768420.533553:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660352:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768420.533571:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768420.533581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768420/real 1775768420] req@ffff9d81901ae680 x1862023066660352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768420.533592:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066660352/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768420.533600:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768420.533603:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768420.533606:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660352:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768420.533611:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660480:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768420.533628:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768420.533637:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768420/real 1775768420] req@ffff9d81901afb80 x1862023066660480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768420.533647:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066660480/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768420.533656:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768420.533659:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768420.533662:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660480:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768420.533667:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660608:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768420.533683:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768420.533694:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768420/real 1775768420] req@ffff9d81901ad180 x1862023066660608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768420.533704:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066660608/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768420.533712:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768420.533714:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768420.533717:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660608:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768421.173043:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768421.556982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768422.197049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768422.581005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768423.226287:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768423.604998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768424.245008:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768424.628976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768425.269090:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768425.652999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768425.653008:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768425.653016:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768425.653022:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768425.653034:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5257 00000100:00080000:3.0:1775768425.653047:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768425.653069:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768425.653117:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066660736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768425.653147:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768425.653151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768425.653153:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768425.653158:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5257 00000100:00080000:3.0:1775768425.653162:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768425.653167:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768425.653177:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066660736:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768425.653179:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768425.653196:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066660864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768425.653206:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768425.653210:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768425.653212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768425.653216:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5257 00000100:00080000:3.0:1775768425.653220:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768425.653224:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768425.653229:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768425.653239:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066660992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768425.653249:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768425.653253:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1775768425.653256:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768425.653256:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768425.653264:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5257 00000100:00080000:3.0:1775768425.653267:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768425.653271:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768425.653277:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768425.653287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066661120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768425.653297:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768425.653300:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768425.653302:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768425.653307:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5257 00000100:00080000:3.0:1775768425.653311:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768425.653315:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768425.653319:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768425.653321:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768425/real 1775768425] req@ffff9d8188740e00 x1862023066660736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768425.653333:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066661248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768425.653344:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066660736/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768425.653353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768425.653357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768425.653360:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066660736:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768425.653384:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660864:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768425.653405:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768425.653418:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768425/real 1775768425] req@ffff9d8188741180 x1862023066660864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768425.653444:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066660864/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768425.653450:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768425.653452:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768425.653454:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660864:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768425.653458:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660992:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768425.653472:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768425.653479:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768425/real 1775768425] req@ffff9d8188743100 x1862023066660992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768425.653486:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066660992/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768425.653490:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768425.653492:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768425.653494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066660992:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768425.653497:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661120:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768425.653515:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768425.653520:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768425/real 1775768425] req@ffff9d8188742d80 x1862023066661120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768425.653527:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066661120/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768425.653532:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768425.653535:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768425.653536:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661120:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768425.653539:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661248:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768425.653548:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768425.653553:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768425/real 1775768425] req@ffff9d8188743b80 x1862023066661248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768425.653559:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066661248/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768425.653564:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768425.653566:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768425.653567:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661248:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768426.293230:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768426.676999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768427.317132:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768427.701031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768428.341148:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768428.725109:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768429.365088:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768429.749036:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768430.389054:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768430.773046:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768430.773059:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768430.773065:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768430.773078:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5262 00000100:00100000:1.0:1775768430.773083:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768430.773092:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768430.773102:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768430.773153:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066661376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768430.773192:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768430.773197:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768430.773199:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768430.773203:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5262 00000100:00080000:3.0:1775768430.773207:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768430.773211:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768430.773223:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768430.773243:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066661504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768430.773254:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768430.773257:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768430.773259:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768430.773263:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5262 00000100:00080000:3.0:1775768430.773267:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768430.773270:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768430.773275:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768430.773284:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066661632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768430.773293:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768430.773297:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768430.773299:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768430.773303:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5262 00000100:00080000:3.0:1775768430.773306:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768430.773310:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768430.773314:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768430.773326:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066661760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768430.773334:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768430.773337:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768430.773340:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768430.773343:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5262 00000100:00080000:3.0:1775768430.773347:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768430.773350:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768430.773354:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768430.773363:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066661888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768430.773441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066661376:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775768430.773512:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768430.773565:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768430/real 1775768430] req@ffff9d8188741500 x1862023066661376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768430.773593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066661376/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768430.773605:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768430.773610:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768430.773614:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066661376:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768430.773633:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661504:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768430.773654:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768430.773666:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768430/real 1775768430] req@ffff9d8188740a80 x1862023066661504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768430.773677:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066661504/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768430.773685:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768430.773688:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768430.773691:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661504:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768430.773697:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661632:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768430.773713:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768430.773722:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768430/real 1775768430] req@ffff9d8188743480 x1862023066661632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768430.773734:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066661632/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768430.773742:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768430.773745:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768430.773749:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661632:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768430.773755:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661760:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768430.773772:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768430.773781:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768430/real 1775768430] req@ffff9d8188743b80 x1862023066661760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768430.773792:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066661760/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768430.773800:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768430.773804:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768430.773806:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661760:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768430.773811:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661888:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768430.773828:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768430.773838:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768430/real 1775768430] req@ffff9d8188742d80 x1862023066661888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768430.773849:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066661888/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768430.773858:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768430.773861:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768430.773865:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066661888:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768431.413896:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768431.797147:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768432.439044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768432.821039:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768433.461106:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768433.846042:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768434.485061:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768434.869016:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768435.509268:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768435.893022:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768435.893029:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768435.893032:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768435.893040:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5267 00000100:00080000:3.0:1775768435.893048:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768435.893097:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768435.893145:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066662016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768435.893178:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768435.893182:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768435.893184:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768435.893188:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5267 00000100:00080000:3.0:1775768435.893191:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768435.893197:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768435.893205:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768435.893218:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066662144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768435.893227:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768435.893231:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768435.893232:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768435.893235:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5267 00000100:00080000:3.0:1775768435.893237:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768435.893240:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768435.893243:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768435.893250:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066662272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768435.893256:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768435.893258:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768435.893259:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768435.893261:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5267 00000100:00080000:3.0:1775768435.893262:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768435.893264:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768435.893266:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768435.893271:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066662400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768435.893276:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768435.893278:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768435.893279:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768435.893281:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5267 00000100:00080000:3.0:1775768435.893283:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768435.893285:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768435.893287:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768435.893291:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066662528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768435.894585:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066662016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768435.894599:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000800:00000100:1.0:1775768435.894682:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768435.894725:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768435/real 1775768435] req@ffff9d8188743100 x1862023066662016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768435.894742:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066662016/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768435.894752:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768435.894756:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768435.894760:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066662016:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768435.894767:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662144:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768435.894788:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768435.894799:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768435/real 1775768435] req@ffff9d8188741180 x1862023066662144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768435.894810:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066662144/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768435.894817:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768435.894820:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768435.894823:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662144:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768435.894829:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662272:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768435.894847:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768435.894858:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768435/real 1775768435] req@ffff9d8188740e00 x1862023066662272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768435.894870:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066662272/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768435.894896:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768435.894899:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768435.894903:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662272:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768435.894909:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662400:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768435.894928:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768435.894939:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768435/real 1775768435] req@ffff9d8188741f80 x1862023066662400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768435.894950:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066662400/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768435.894971:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768435.894975:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768435.894978:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662400:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768435.894984:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662528:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768435.895003:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768435.895013:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768435/real 1775768435] req@ffff9d8188742680 x1862023066662528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768435.895024:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066662528/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768435.895033:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768435.895036:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768435.895039:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662528:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768436.533144:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768436.917006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768437.557065:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768437.940973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768438.581107:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768438.965004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768439.605035:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768439.989006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768440.629069:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768441.012985:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768441.012995:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768441.012999:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1775768441.013006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768441.013024:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5272 00000100:00080000:3.0:1775768441.013035:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768441.013041:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768441.013076:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066662656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768441.013104:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768441.013108:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768441.013109:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768441.013113:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5272 00000100:00080000:3.0:1775768441.013117:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768441.013121:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768441.013130:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768441.013144:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066662784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768441.013155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768441.013159:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1775768441.013161:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066662656:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768441.013162:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768441.013170:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5272 00000100:00080000:3.0:1775768441.013174:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768441.013177:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768441.013181:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768441.013191:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066662912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768441.013200:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768441.013203:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768441.013205:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768441.013209:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5272 00000100:00080000:3.0:1775768441.013212:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768441.013216:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768441.013221:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768441.013232:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066663040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768441.013246:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1775768441.013249:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768441.013251:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768441.013254:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768441.013259:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5272 00000100:00080000:3.0:1775768441.013263:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768441.013267:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768441.013271:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768441.013282:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066663168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768441.013298:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768441/real 1775768441] req@ffff9d8188741880 x1862023066662656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768441.013317:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066662656/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768441.013327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768441.013331:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768441.013335:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066662656:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768441.013355:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662784:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768441.013382:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768441.013393:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768441/real 1775768441] req@ffff9d8188743800 x1862023066662784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768441.013405:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066662784/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768441.013412:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768441.013416:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768441.013419:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662784:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768441.013425:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662912:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768441.013444:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768441.013454:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768441/real 1775768441] req@ffff9d8188742300 x1862023066662912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768441.013465:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066662912/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768441.013472:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768441.013475:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768441.013478:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066662912:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768441.013484:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663040:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768441.013502:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768441.013511:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768441/real 1775768441] req@ffff9d8188742a00 x1862023066663040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768441.013523:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066663040/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768441.013542:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768441.013545:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768441.013548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663040:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768441.013554:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663168:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768441.013572:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768441.013582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768441/real 1775768441] req@ffff9d8188740700 x1862023066663168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768441.013594:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066663168/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768441.013601:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768441.013604:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768441.013607:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663168:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768441.653084:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768442.036995:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768442.677179:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768443.061012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768443.702317:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768444.085008:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768444.725067:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768445.108981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768445.749070:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768446.132997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768446.133033:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768446.133042:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768446.133046:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768446.133056:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5277 00000100:00080000:0.0:1775768446.133070:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768446.133078:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768446.133129:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066663296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768446.133155:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768446.133158:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768446.133159:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768446.133162:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5277 00000100:00080000:0.0:1775768446.133165:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768446.133168:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768446.133178:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768446.133189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066663424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768446.133208:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768446.133211:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768446.133214:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768446.133219:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066663296:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768446.133222:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5277 00000100:00080000:0.0:1775768446.133228:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768446.133233:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768446.133240:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768446.133252:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066663552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768446.133259:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768446.133262:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768446.133263:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768446.133266:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5277 00000100:00080000:0.0:1775768446.133269:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768446.133271:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768446.133280:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768446.133291:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066663680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768446.133299:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768446.133303:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768446.133304:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768446.133310:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5277 00000800:00000100:1.0:1775768446.133311:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775768446.133317:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768446.133322:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768446.133326:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768446.133338:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066663808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768446.133386:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768446/real 1775768446] req@ffff9d81901ad180 x1862023066663296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768446.133406:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066663296/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768446.133415:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768446.133420:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768446.133423:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066663296:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768446.133448:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663424:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768446.133465:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768446.133483:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768446/real 1775768446] req@ffff9d81901afb80 x1862023066663424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768446.133491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066663424/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768446.133501:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768446.133503:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768446.133505:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663424:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768446.133509:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663552:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768446.133520:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768446.133525:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768446/real 1775768446] req@ffff9d81901ae680 x1862023066663552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768446.133534:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066663552/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768446.133542:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768446.133545:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768446.133548:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663552:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768446.133554:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663680:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768446.133572:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768446.133582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768446/real 1775768446] req@ffff9d81901af100 x1862023066663680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768446.133593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066663680/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768446.133601:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768446.133605:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768446.133608:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663680:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768446.133614:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663808:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768446.133632:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768446.133641:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768446/real 1775768446] req@ffff9d81901aed80 x1862023066663808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768446.133662:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066663808/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768446.133671:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768446.133674:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768446.133677:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066663808:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768446.773329:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768447.157105:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768447.797064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768448.180973:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768448.821057:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768449.204990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768449.845198:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768450.228998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768450.869234:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768451.253009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768451.253042:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768451.253047:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768451.253052:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768451.253060:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5282 00000100:00080000:0.0:1775768451.253070:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768451.253076:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768451.253111:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066663936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768451.253135:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768451.253138:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768451.253140:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768451.253143:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5282 00000100:00080000:0.0:1775768451.253146:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768451.253150:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768451.253159:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768451.253170:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066664064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768451.253179:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768451.253182:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768451.253184:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768451.253187:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5282 00000100:00080000:0.0:1775768451.253191:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768451.253194:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768451.253198:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768451.253207:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066664192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768451.253220:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768451.253223:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768451.253224:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768451.253242:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066663936:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768451.253288:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5282 00000100:00080000:0.0:1775768451.253292:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1775768451.253294:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:0.0:1775768451.253296:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768451.253301:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768451.253310:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066664320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768451.253318:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768451.253321:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768451.253323:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768451.253326:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5282 00000100:00080000:0.0:1775768451.253328:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768451.253331:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768451.253334:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768451.253334:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768451/real 1775768451] req@ffff9d81901aed80 x1862023066663936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768451.253341:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066664448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768451.253348:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066663936/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768451.253357:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768451.253361:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768451.253364:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066663936:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768451.253370:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664064:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768451.253389:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768451.253398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768451/real 1775768451] req@ffff9d81901af100 x1862023066664064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768451.253405:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066664064/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768451.253410:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768451.253412:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768451.253414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664064:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768451.253417:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664192:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768451.253426:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768451.253431:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768451/real 1775768451] req@ffff9d81901ae680 x1862023066664192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768451.253437:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066664192/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768451.253441:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768451.253443:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768451.253444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664192:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768451.253460:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664320:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768451.253469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768451.253474:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768451/real 1775768451] req@ffff9d81901afb80 x1862023066664320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768451.253480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066664320/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768451.253486:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768451.253487:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768451.253489:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664320:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768451.253492:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664448:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768451.253500:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768451.253504:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768451/real 1775768451] req@ffff9d81901ad180 x1862023066664448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768451.253510:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066664448/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768451.253515:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768451.253516:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768451.253518:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664448:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768451.893097:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768452.277926:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768452.917109:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768453.302229:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768453.941143:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768454.324992:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768454.965827:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768455.348969:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768455.989125:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768456.373534:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768456.373541:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768456.373546:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768456.373554:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5287 00000100:00080000:3.0:1775768456.373563:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768456.373568:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768456.373599:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066664576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768456.373624:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768456.373627:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768456.373628:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768456.373630:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5287 00000100:00080000:3.0:1775768456.373632:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768456.373634:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768456.373641:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768456.373648:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066664704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768456.373654:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768456.373656:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768456.373657:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768456.373659:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5287 00000100:00080000:3.0:1775768456.373660:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768456.373662:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768456.373664:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768456.373669:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066664832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768456.373679:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768456.373681:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768456.373682:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768456.373684:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5287 00000100:00080000:3.0:1775768456.373685:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768456.373687:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768456.373689:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768456.373695:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066664960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768456.373699:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768456.373701:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768456.373702:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768456.373703:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5287 00000100:00080000:3.0:1775768456.373705:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768456.373706:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768456.373709:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768456.373714:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066665088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768456.373797:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066664576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768456.373868:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768456.373898:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768456.373908:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768456.373918:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665088:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768456.374009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768457.013022:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768457.396997:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768458.037069:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768458.421006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768459.061059:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768459.444980:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775768459.445094:0:10934:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775768459.445127:0:10934:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775768459.445144:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768459.445192:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768459.445202:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768459.445209:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768459.445217:0:10934:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775768459.445282:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768456/real 1775768459] req@ffff9d8188740380 x1862023066664576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768459.445307:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066664576/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768459.445318:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768459.445322:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768459.445326:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066664576:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768459.445333:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768456/real 1775768459] req@ffff9d8188741c00 x1862023066664704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768459.445345:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066664704/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768459.445353:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768459.445357:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768459.445360:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664704:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768459.445366:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768456/real 1775768459] req@ffff9d8188740000 x1862023066664832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768459.445376:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066664832/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768459.445383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768459.445386:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768459.445389:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664832:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768459.445395:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768456/real 1775768459] req@ffff9d8188740700 x1862023066664960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768459.445409:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066664960/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768459.445418:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768459.445422:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768459.445425:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066664960:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768459.445430:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768456/real 1775768459] req@ffff9d8188742a00 x1862023066665088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768459.445439:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066665088/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768459.445446:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768459.445449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768459.445452:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665088:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768460.085042:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768460.469000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768461.109133:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768461.493005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768461.493007:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768461.493019:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768461.493025:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768461.493038:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5293 00000100:00080000:3.0:1775768461.493053:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768461.493060:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768461.493106:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066665216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768461.493149:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768461.493153:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768461.493155:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768461.493159:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5293 00000100:00080000:3.0:1775768461.493163:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768461.493167:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768461.493179:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768461.493192:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066665344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768461.493201:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768461.493205:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768461.493207:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768461.493211:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5293 00000100:00080000:3.0:1775768461.493214:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768461.493218:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768461.493223:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768461.493234:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066665472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768461.493242:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768461.493245:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768461.493247:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768461.493251:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5293 00000100:00080000:3.0:1775768461.493254:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768461.493257:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768461.493261:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768461.493272:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066665600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768461.493280:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768461.493284:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768461.493285:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768461.493289:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5293 00000100:00080000:3.0:1775768461.493292:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768461.493296:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768461.493300:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768461.493310:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066665728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768461.493427:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066665216:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775768461.493517:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768461.493578:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768461/real 1775768461] req@ffff9d8188742300 x1862023066665216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768461.493598:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066665216/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768461.493610:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768461.493614:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768461.493618:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066665216:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768461.493626:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665344:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768461.493649:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768461.493661:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768461/real 1775768461] req@ffff9d8188743800 x1862023066665344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768461.493672:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066665344/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768461.493680:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768461.493683:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768461.493686:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665344:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768461.493692:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665472:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768461.493727:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768461.493739:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768461/real 1775768461] req@ffff9d8188741880 x1862023066665472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768461.493749:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066665472/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768461.493757:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768461.493761:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768461.493764:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665472:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768461.493771:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665600:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768461.493790:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768461.493800:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768461/real 1775768461] req@ffff9d8188742680 x1862023066665600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768461.493812:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066665600/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768461.493822:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768461.493825:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768461.493829:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665600:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768461.493834:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665728:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768461.493852:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768461.493860:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768461/real 1775768461] req@ffff9d8188741f80 x1862023066665728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768461.493873:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066665728/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768461.493910:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768461.493914:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768461.493917:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665728:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768462.133478:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768462.517011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768463.157143:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768463.541011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768464.181040:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768464.565000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768465.205319:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768465.589005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768466.229183:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768466.612940:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768466.612949:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768466.612953:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768466.612962:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5298 00000100:00080000:3.0:1775768466.612972:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768466.612978:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768466.613011:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066665856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768466.613035:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768466.613039:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768466.613040:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768466.613044:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5298 00000100:00080000:3.0:1775768466.613047:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768466.613050:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768466.613058:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768466.613068:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066665984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768466.613077:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768466.613080:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768466.613082:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768466.613085:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5298 00000100:00080000:3.0:1775768466.613088:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768466.613091:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768466.613096:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768466.613104:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066666112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768466.613111:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768466.613114:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768466.613115:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768466.613119:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5298 00000100:00080000:3.0:1775768466.613121:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768466.613124:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768466.613128:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768466.613135:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066666240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768466.613142:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768466.613145:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768466.613147:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768466.613149:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5298 00000100:00080000:3.0:1775768466.613152:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768466.613155:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768466.613158:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768466.613166:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066666368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768466.613744:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066665856:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775768466.613801:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768466.613838:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768466/real 1775768466] req@ffff9d8188740e00 x1862023066665856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768466.613852:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066665856/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768466.613859:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768466.613863:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768466.613865:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066665856:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768466.613870:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665984:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768466.613900:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768466.613908:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768466/real 1775768466] req@ffff9d8188741180 x1862023066665984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768466.613915:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066665984/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768466.613920:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768466.613922:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768466.613924:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066665984:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768466.613927:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666112:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768466.613938:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768466.613947:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768466/real 1775768466] req@ffff9d8188743100 x1862023066666112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768466.613957:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066666112/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768466.613965:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768466.613968:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768466.613971:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666112:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768466.613976:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666240:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768466.613994:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768466.614005:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768466/real 1775768466] req@ffff9d8188742d80 x1862023066666240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768466.614015:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066666240/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768466.614024:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768466.614027:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768466.614030:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666240:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768466.614036:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666368:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768466.614049:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768466.614055:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768466/real 1775768466] req@ffff9d8188743b80 x1862023066666368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768466.614062:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066666368/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768466.614067:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768466.614069:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768466.614071:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666368:192.168.202.105@tcp:8:kworker.0 00000100:00100000:2.0:1775768466.615965:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768467.255587:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768467.637005:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768468.278511:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768468.661051:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768469.304041:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768469.685045:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768470.326512:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768470.709011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768471.349212:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768471.733021:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768471.733082:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768471.733089:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768471.733093:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768471.733101:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5303 00000100:00080000:0.0:1775768471.733112:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768471.733119:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768471.733157:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066666496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768471.733188:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768471.733192:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768471.733194:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768471.733197:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5303 00000100:00080000:0.0:1775768471.733200:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768471.733203:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768471.733212:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768471.733222:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066666624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768471.733231:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768471.733234:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768471.733235:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768471.733239:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5303 00000100:00080000:0.0:1775768471.733241:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768471.733244:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768471.733247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768471.733256:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066666752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768471.733264:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768471.733267:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768471.733268:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768471.733271:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5303 00000100:00080000:0.0:1775768471.733273:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768471.733276:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768471.733280:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:0.0:1775768471.733287:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066666880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768471.733294:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768471.733297:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768471.733298:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768471.733301:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5303 00000100:00080000:0.0:1775768471.733303:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768471.733306:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768471.733309:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768471.733316:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066667008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768471.733395:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066666496:192.168.202.105@tcp:250:kworker.0 00000800:00000100:1.0:1775768471.733451:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768471.733495:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768471/real 1775768471] req@ffff9d81901ad180 x1862023066666496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768471.733512:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066666496/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768471.733521:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768471.733524:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768471.733540:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066666496:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768471.733548:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666624:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768471.733571:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768471.733582:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768471/real 1775768471] req@ffff9d81901afb80 x1862023066666624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768471.733593:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066666624/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768471.733600:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768471.733603:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768471.733605:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666624:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768471.733624:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666752:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768471.733654:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768471.733664:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768471/real 1775768471] req@ffff9d81901ae680 x1862023066666752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768471.733675:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066666752/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768471.733682:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768471.733685:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768471.733688:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666752:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768471.733693:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666880:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768471.733712:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768471.733722:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768471/real 1775768471] req@ffff9d81901af100 x1862023066666880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768471.733733:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066666880/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768471.733741:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768471.733744:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768471.733746:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066666880:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768471.733752:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667008:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768471.733769:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768471.733777:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768471/real 1775768471] req@ffff9d81901aed80 x1862023066667008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768471.733788:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066667008/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768471.733796:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768471.733799:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768471.733802:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667008:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768472.373064:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768472.757476:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768473.397861:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768473.783280:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768474.421097:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768474.804998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768475.446045:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768475.829004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768476.469060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768476.853006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768476.853134:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768476.853141:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768476.853145:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768476.853154:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5308 00000100:00080000:3.0:1775768476.853163:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768476.853170:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768476.853203:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743480 x1862023066667136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768476.853229:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768476.853232:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768476.853234:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768476.853238:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5308 00000100:00080000:3.0:1775768476.853242:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768476.853245:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768476.853253:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768476.853265:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740a80 x1862023066667264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768476.853276:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768476.853281:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066667136:192.168.202.105@tcp:250:kworker.0 00000100:00080000:3.0:1775768476.853281:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768476.853283:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768476.853290:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5308 00000100:00080000:3.0:1775768476.853293:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768476.853296:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768476.853301:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768476.853311:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741500 x1862023066667392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768476.853320:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768476.853323:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768476.853325:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768476.853328:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5308 00000100:00080000:3.0:1775768476.853332:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768476.853335:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768476.853339:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775768476.853342:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768476.853352:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743b80 x1862023066667520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768476.853362:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768476.853365:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768476.853367:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768476.853371:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5308 00000100:00080000:3.0:1775768476.853375:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768476.853379:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768476.853385:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768476.853388:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768476/real 1775768476] req@ffff9d8188743480 x1862023066667136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768476.853397:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742d80 x1862023066667648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768476.853410:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743480 x1862023066667136/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768476.853421:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768476.853425:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768476.853430:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066667136:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768476.853449:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667264:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768476.853469:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768476.853481:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768476/real 1775768476] req@ffff9d8188740a80 x1862023066667264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768476.853493:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740a80 x1862023066667264/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768476.853506:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768476.853509:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768476.853512:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667264:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768476.853518:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667392:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768476.853540:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768476.853551:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768476/real 1775768476] req@ffff9d8188741500 x1862023066667392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768476.853578:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741500 x1862023066667392/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768476.853585:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768476.853588:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768476.853591:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667392:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768476.853596:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667520:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768476.853615:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768476.853624:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768476/real 1775768476] req@ffff9d8188743b80 x1862023066667520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768476.853633:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743b80 x1862023066667520/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768476.853641:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768476.853643:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768476.853647:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667520:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768476.853652:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667648:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768476.853668:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768476.853677:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768476/real 1775768476] req@ffff9d8188742d80 x1862023066667648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768476.853688:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742d80 x1862023066667648/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768476.853695:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768476.853698:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768476.853700:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667648:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768477.493117:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768477.876999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768478.517059:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1775768478.901011:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768479.541105:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768479.925032:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768480.565021:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768480.949056:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768481.589068:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768481.973010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:1.0:1775768481.973066:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768481.973072:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1775768481.973077:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768481.973088:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5313 00000100:00080000:1.0:1775768481.973099:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768481.973106:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768481.973164:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553480 x1862023066667776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768481.973196:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768481.973200:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1775768481.973202:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768481.973206:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5313 00000100:00080000:1.0:1775768481.973210:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768481.973214:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768481.973225:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768481.973238:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190550e00 x1862023066667904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768481.973247:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768481.973250:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1775768481.973252:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768481.973256:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5313 00000100:00080000:1.0:1775768481.973259:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768481.973263:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768481.973267:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768481.973277:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190553100 x1862023066668032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1775768481.973286:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768481.973289:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1775768481.973291:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768481.973295:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5313 00000100:00080000:1.0:1775768481.973305:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768481.973309:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768481.973315:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768481.973328:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551500 x1862023066668160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768481.973336:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066667776:192.168.202.105@tcp:250:kworker.0 00000100:00080000:1.0:1775768481.973341:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1775768481.973345:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1775768481.973347:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768481.973351:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5313 00000100:00080000:1.0:1775768481.973354:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1775768481.973358:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:1.0:1775768481.973362:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768481.973371:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8190551880 x1862023066668288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768481.973416:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768481.973461:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768481/real 1775768481] req@ffff9d8190553480 x1862023066667776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768481.973480:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553480 x1862023066667776/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768481.973495:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768481.973499:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768481.973504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066667776:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768481.973541:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667904:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768481.973562:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768481.973573:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768481/real 1775768481] req@ffff9d8190550e00 x1862023066667904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768481.973585:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190550e00 x1862023066667904/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768481.973592:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768481.973596:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768481.973598:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066667904:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768481.973604:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668032:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768481.973621:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768481.973641:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768481/real 1775768481] req@ffff9d8190553100 x1862023066668032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768481.973652:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190553100 x1862023066668032/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768481.973660:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768481.973662:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768481.973666:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668032:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768481.973691:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668160:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768481.973708:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768481.973717:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768481/real 1775768481] req@ffff9d8190551500 x1862023066668160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768481.973726:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551500 x1862023066668160/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768481.973734:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768481.973737:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768481.973740:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668160:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768481.973744:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668288:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768481.973758:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768481.973767:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768481/real 1775768481] req@ffff9d8190551880 x1862023066668288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768481.973777:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8190551880 x1862023066668288/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768481.973785:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768481.973788:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768481.973791:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8190551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668288:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768482.613157:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768482.997013:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768483.637107:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768484.020994:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768484.661102:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768485.045004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768485.685035:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768486.068990:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768486.709049:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768487.092994:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768487.092999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768487.093004:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768487.093009:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768487.093019:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5318 00000100:00080000:2.0:1775768487.093029:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768487.093036:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768487.093070:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066668416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768487.093097:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768487.093100:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768487.093102:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768487.093106:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5318 00000100:00080000:2.0:1775768487.093109:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768487.093113:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768487.093122:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768487.093133:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066668416:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768487.093147:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066668544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768487.093158:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768487.093162:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768487.093163:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768487.093168:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5318 00000100:00080000:2.0:1775768487.093179:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768487.093183:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000800:00000100:0.0:1775768487.093189:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000008:00100000:2.0:1775768487.093189:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768487.093202:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066668672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768487.093212:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768487.093216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768487.093218:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768487.093222:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5318 00000100:00080000:2.0:1775768487.093226:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768487.093230:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768487/real 1775768487] req@ffff9d81984a1180 x1862023066668416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768487.093233:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768487.093240:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768487.093253:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066668800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768487.093256:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066668416/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768487.093271:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768487.093275:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768487.093276:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768487.093277:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768487.093285:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775768487.093286:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5318 00000100:00080000:2.0:1775768487.093292:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1775768487.093293:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066668416:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768487.093300:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768487.093304:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775768487.093314:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066668928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768487.093319:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668544:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768487.093344:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768487.093353:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768487/real 1775768487] req@ffff9d81984a1f80 x1862023066668544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768487.093361:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066668544/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768487.093369:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768487.093372:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768487.093375:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668544:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768487.093381:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668672:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768487.093398:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768487.093408:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768487/real 1775768487] req@ffff9d81984a0e00 x1862023066668672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768487.093419:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066668672/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768487.093429:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768487.093431:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768487.093433:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668672:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768487.093437:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668800:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768487.093450:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768487.093469:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768487/real 1775768487] req@ffff9d81984a2680 x1862023066668800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768487.093477:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066668800/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768487.093483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768487.093485:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768487.093487:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668800:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768487.093496:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668928:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768487.093508:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768487.093514:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768487/real 1775768487] req@ffff9d81984a0000 x1862023066668928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768487.093520:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066668928/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768487.093525:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768487.093527:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768487.093529:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066668928:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768487.733022:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768488.117031:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768488.757094:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768489.141028:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768489.780984:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768490.164985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768490.805038:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768491.189004:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768491.828995:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768492.212960:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768492.212968:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768492.212968:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768492.212973:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768492.212982:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5323 00000100:00080000:2.0:1775768492.212991:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768492.212998:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768492.213032:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066669056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768492.213058:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768492.213061:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768492.213062:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768492.213065:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5323 00000100:00080000:2.0:1775768492.213068:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768492.213071:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768492.213079:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768492.213089:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066669184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768492.213098:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1775768492.213101:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066669056:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768492.213102:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768492.213113:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768492.213118:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5323 00000100:00080000:2.0:1775768492.213125:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768492.213128:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768492.213133:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768492.213142:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066669312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1775768492.213150:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775768492.213155:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768492.213158:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768492.213160:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768492.213164:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5323 00000100:00080000:2.0:1775768492.213167:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768492.213170:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768492.213174:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:2.0:1775768492.213186:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066669440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768492.213189:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768492/real 1775768492] req@ffff9d81984a0700 x1862023066669056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768492.213207:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768492.213210:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768492.213212:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1775768492.213213:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066669056/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768492.213220:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5323 00000100:00080000:0.0:1775768492.213225:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768492.213225:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768492.213233:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775768492.213234:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768492.213239:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768492.213251:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066669056:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768492.213251:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066669568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768492.213275:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669184:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768492.213295:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768492.213306:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768492/real 1775768492] req@ffff9d81984a2a00 x1862023066669184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768492.213318:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066669184/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768492.213327:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768492.213330:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768492.213333:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669184:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768492.213337:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669312:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768492.213350:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768492.213356:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768492/real 1775768492] req@ffff9d81984a3480 x1862023066669312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768492.213363:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066669312/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768492.213368:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768492.213370:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768492.213371:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669312:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768492.213374:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669440:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768492.213385:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768492.213402:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768492/real 1775768492] req@ffff9d81984a2d80 x1862023066669440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768492.213425:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066669440/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768492.213433:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768492.213436:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768492.213439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669440:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768492.213445:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669568:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768492.213470:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768492.213478:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768492/real 1775768492] req@ffff9d81984a3b80 x1862023066669568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768492.213485:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066669568/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768492.213491:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768492.213493:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768492.213494:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669568:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768492.853047:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768493.236955:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768493.877000:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768494.260998:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768494.901060:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768495.285053:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768495.925084:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768496.309012:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768496.949184:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768497.332943:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768497.333058:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768497.333062:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768497.333065:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768497.333071:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5328 00000100:00080000:3.0:1775768497.333079:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768497.333084:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768497.333113:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743100 x1862023066669696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768497.333141:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768497.333145:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768497.333146:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768497.333150:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5328 00000100:00080000:3.0:1775768497.333154:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768497.333158:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768497.333167:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768497.333174:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066669696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:3.0:1775768497.333180:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741180 x1862023066669824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768497.333188:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768497.333190:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768497.333191:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768497.333194:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5328 00000100:00080000:3.0:1775768497.333196:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768497.333199:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768497.333203:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768497.333210:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740e00 x1862023066669952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768497.333216:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768497.333218:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768497.333219:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768497.333229:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5328 00000800:00000100:0.0:1775768497.333233:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:3.0:1775768497.333233:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768497.333240:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768497.333245:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768497.333257:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741f80 x1862023066670080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768497.333267:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768497.333270:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768497.333272:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768497.333276:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5328 00000100:00100000:0.0:1775768497.333279:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768497/real 1775768497] req@ffff9d8188743100 x1862023066669696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1775768497.333282:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768497.333289:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768497.333294:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768497.333303:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743100 x1862023066669696/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1775768497.333304:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742680 x1862023066670208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768497.333334:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768497.333339:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768497.333344:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066669696:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768497.333391:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669824:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768497.333413:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768497.333424:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768497/real 1775768497] req@ffff9d8188741180 x1862023066669824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768497.333436:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741180 x1862023066669824/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768497.333444:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768497.333447:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768497.333451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669824:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768497.333457:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669952:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768497.333471:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768497.333480:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768497/real 1775768497] req@ffff9d8188740e00 x1862023066669952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768497.333491:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740e00 x1862023066669952/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768497.333498:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768497.333501:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768497.333504:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066669952:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768497.333509:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670080:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768497.333526:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768497.333535:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768497/real 1775768497] req@ffff9d8188741f80 x1862023066670080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768497.333545:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741f80 x1862023066670080/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768497.333554:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768497.333556:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768497.333559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670080:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768497.333564:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670208:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768497.333579:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768497.333587:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768497/real 1775768497] req@ffff9d8188742680 x1862023066670208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768497.333597:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742680 x1862023066670208/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768497.333603:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768497.333606:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768497.333608:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670208:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768497.973000:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768498.357009:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768498.997035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768499.380949:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768500.021062:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768500.405001:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768501.045035:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768501.428987:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768502.069034:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1775768502.452969:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1775768502.452971:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768502.452977:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768502.452980:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768502.452988:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5333 00000100:00080000:3.0:1775768502.452997:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768502.453003:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768502.453036:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741880 x1862023066670336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768502.453058:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768502.453060:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768502.453061:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768502.453064:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5333 00000100:00080000:3.0:1775768502.453066:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768502.453068:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768502.453075:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768502.453082:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188743800 x1862023066670464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768502.453089:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768502.453091:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768502.453092:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768502.453094:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5333 00000100:00080000:3.0:1775768502.453096:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768502.453100:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768502.453101:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066670336:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768502.453103:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768502.453122:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742300 x1862023066670592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768502.453128:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768502.453131:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768502.453132:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768502.453135:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5333 00000100:00080000:3.0:1775768502.453149:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768502.453151:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768502.453155:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:0.0:1775768502.453158:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:3.0:1775768502.453164:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066670720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768502.453171:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768502.453173:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768502.453174:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768502.453175:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5333 00000100:00080000:3.0:1775768502.453177:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768502.453179:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768502.453182:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768502.453189:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066670848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1775768502.453203:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768502/real 1775768502] req@ffff9d8188741880 x1862023066670336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768502.453222:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741880 x1862023066670336/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768502.453232:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768502.453236:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768502.453240:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066670336:192.168.202.105@tcp:250:kworker.0 00000100:00100000:0.0:1775768502.453248:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670464:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768502.453265:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768502.453273:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768502/real 1775768502] req@ffff9d8188743800 x1862023066670464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768502.453281:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188743800 x1862023066670464/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768502.453289:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768502.453292:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768502.453295:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188743800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670464:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768502.453341:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670592:192.168.202.105@tcp:38:kworker.0 00000800:00000100:0.0:1775768502.453355:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768502.453361:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768502/real 1775768502] req@ffff9d8188742300 x1862023066670592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768502.453368:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742300 x1862023066670592/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768502.453373:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768502.453375:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768502.453377:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670592:192.168.202.105@tcp:38:kworker.0 00000100:00100000:0.0:1775768502.453380:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670720:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768502.453390:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768502.453395:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768502/real 1775768502] req@ffff9d8188742a00 x1862023066670720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768502.453401:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066670720/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768502.453406:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768502.453407:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768502.453409:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670720:192.168.202.105@tcp:8:kworker.0 00000100:00100000:0.0:1775768502.453412:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670848:192.168.202.105@tcp:8:kworker.0 00000800:00000100:0.0:1775768502.453421:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768502.453425:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768502/real 1775768502] req@ffff9d8188740700 x1862023066670848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1775768502.453431:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066670848/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1775768502.453436:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1775768502.453437:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:0.0:1775768502.453439:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066670848:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768503.093020:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768503.476951:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768504.117079:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768504.500976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768505.141051:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768505.524942:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768506.165053:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768506.549003:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768507.189115:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768507.572993:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768507.573319:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768507.573325:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768507.573329:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768507.573339:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5339 00000100:00080000:2.0:1775768507.573348:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768507.573355:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768507.573388:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2300 x1862023066670976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768507.573417:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768507.573421:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768507.573423:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768507.573426:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5339 00000100:00080000:2.0:1775768507.573430:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768507.573434:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768507.573443:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768507.573457:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3b80 x1862023066671104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768507.573463:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066670976:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768507.573472:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768507.573476:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768507.573477:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768507.573481:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5339 00000100:00080000:2.0:1775768507.573485:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768507.573489:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768507.573493:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768507.573505:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2d80 x1862023066671232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768507.573514:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768507.573516:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768507.573517:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768507.573520:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5339 00000100:00080000:2.0:1775768507.573524:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768507.573527:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768507.573530:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775768507.573534:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775768507.573541:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3480 x1862023066671360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768507.573547:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768507.573549:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768507.573550:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768507.573552:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5339 00000100:00080000:2.0:1775768507.573554:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768507.573556:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768507.573558:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:2.0:1775768507.573567:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2a00 x1862023066671488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768507.573576:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768507/real 1775768507] req@ffff9d81984a2300 x1862023066670976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768507.573589:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2300 x1862023066670976/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768507.573595:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768507.573598:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768507.573600:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066670976:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768507.573617:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671104:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768507.573629:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768507.573636:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768507/real 1775768507] req@ffff9d81984a3b80 x1862023066671104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768507.573643:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3b80 x1862023066671104/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768507.573648:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768507.573649:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768507.573652:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671104:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768507.573656:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671232:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768507.573664:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768507.573669:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768507/real 1775768507] req@ffff9d81984a2d80 x1862023066671232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768507.573674:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2d80 x1862023066671232/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768507.573679:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768507.573680:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768507.573682:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671232:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768507.573685:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671360:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768507.573694:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768507.573698:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768507/real 1775768507] req@ffff9d81984a3480 x1862023066671360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768507.573704:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3480 x1862023066671360/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768507.573709:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768507.573711:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768507.573712:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671360:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768507.573715:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671488:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768507.573723:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768507.573727:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768507/real 1775768507] req@ffff9d81984a2a00 x1862023066671488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768507.573734:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2a00 x1862023066671488/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768507.573738:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768507.573739:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768507.573741:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671488:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768508.213009:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768508.597296:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768509.237030:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768509.621534:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768510.261050:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768510.645000:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768511.285010:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768511.668985:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768512.309054:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768512.692999:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:0.0:1775768512.693012:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768512.693023:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1775768512.693028:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768512.693038:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5344 00000100:00080000:0.0:1775768512.693051:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768512.693058:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:0.0:1775768512.693108:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901aed80 x1862023066671616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768512.693147:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768512.693151:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1775768512.693153:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768512.693157:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5344 00000100:00080000:0.0:1775768512.693161:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768512.693166:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768512.693178:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768512.693191:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901af100 x1862023066671744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768512.693213:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768512.693216:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1775768512.693218:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768512.693225:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5344 00000100:00100000:1.0:1775768512.693226:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066671616:192.168.202.105@tcp:250:kworker.0 00000100:00080000:0.0:1775768512.693229:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768512.693241:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768512.693247:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1775768512.693263:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ae680 x1862023066671872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768512.693272:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768512.693276:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1775768512.693278:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768512.693282:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5344 00000100:00080000:0.0:1775768512.693295:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768512.693300:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768512.693305:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000800:00000100:1.0:1775768512.693307:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:0.0:1775768512.693324:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901afb80 x1862023066672000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1775768512.693331:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1775768512.693334:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1775768512.693336:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1775768512.693339:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5344 00000100:00080000:0.0:1775768512.693344:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1775768512.693347:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:0.0:1775768512.693351:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:0.0:1775768512.693359:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81901ad180 x1862023066672128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768512.693364:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768512/real 1775768512] req@ffff9d81901aed80 x1862023066671616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768512.693389:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901aed80 x1862023066671616/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768512.693400:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768512.693405:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768512.693408:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066671616:192.168.202.105@tcp:250:kworker.0 00000100:00100000:1.0:1775768512.693441:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671744:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768512.693464:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768512.693476:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768512/real 1775768512] req@ffff9d81901af100 x1862023066671744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768512.693487:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901af100 x1862023066671744/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768512.693493:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768512.693497:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768512.693509:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671744:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768512.693515:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671872:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768512.693532:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768512.693541:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768512/real 1775768512] req@ffff9d81901ae680 x1862023066671872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768512.693550:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ae680 x1862023066671872/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768512.693555:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768512.693557:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768512.693559:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066671872:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768512.693563:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672000:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768512.693576:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768512.693581:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768512/real 1775768512] req@ffff9d81901afb80 x1862023066672000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768512.693587:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901afb80 x1862023066672000/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768512.693593:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768512.693595:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768512.693597:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672000:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768512.693600:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672128:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768512.693609:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768512.693613:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768512/real 1775768512] req@ffff9d81901ad180 x1862023066672128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768512.693619:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81901ad180 x1862023066672128/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768512.693627:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768512.693630:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768512.693633:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81901ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672128:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768513.333072:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768513.716982:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768514.357032:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768514.740967:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768515.381010:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768515.764983:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768516.405059:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768516.788979:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768517.429006:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768517.812984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768517.812997:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768517.813002:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768517.813005:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768517.813013:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5349 00000100:00080000:2.0:1775768517.813021:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768517.813027:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768517.813054:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0700 x1862023066672256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768517.813080:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768517.813086:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768517.813088:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768517.813092:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5349 00000100:00080000:2.0:1775768517.813095:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768517.813099:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768517.813122:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768517.813124:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066672256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768517.813138:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0000 x1862023066672384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768517.813160:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768517.813164:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768517.813166:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768517.813172:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5349 00000100:00080000:2.0:1775768517.813185:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768517.813189:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768517.813194:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1775768517.813195:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:2.0:1775768517.813209:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a2680 x1862023066672512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768517.813218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768517.813221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768517.813223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768517.813226:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5349 00000100:00080000:2.0:1775768517.813230:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768517.813234:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768517.813240:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768517.813251:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768517/real 1775768517] req@ffff9d81984a0700 x1862023066672256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768517.813255:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a0e00 x1862023066672640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768517.813277:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768517.813280:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768517.813282:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1775768517.813285:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0700 x1862023066672256/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768517.813288:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5349 00000100:00080000:2.0:1775768517.813294:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768517.813301:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00080000:1.0:1775768517.813302:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1775768517.813307:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00080000:1.0:1775768517.813309:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768517.813315:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066672256:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768517.813322:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1f80 x1862023066672768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768517.813338:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672384:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768517.813364:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768517.813376:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768517/real 1775768517] req@ffff9d81984a0000 x1862023066672384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768517.813387:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0000 x1862023066672384/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768517.813409:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768517.813412:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768517.813414:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672384:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768517.813418:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672512:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768517.813433:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768517.813440:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768517/real 1775768517] req@ffff9d81984a2680 x1862023066672512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768517.813446:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a2680 x1862023066672512/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768517.813451:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768517.813453:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768517.813455:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672512:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768517.813458:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672640:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768517.813467:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768517.813472:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768517/real 1775768517] req@ffff9d81984a0e00 x1862023066672640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768517.813477:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a0e00 x1862023066672640/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768517.813483:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768517.813484:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768517.813486:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672640:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768517.813488:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672768:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768517.813497:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768517.813502:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768517/real 1775768517] req@ffff9d81984a1f80 x1862023066672768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768517.813508:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1f80 x1862023066672768/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768517.813512:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768517.813514:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768517.813515:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066672768:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768518.453081:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768518.836984:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768519.476997:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768519.861052:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768520.501077:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768520.884981:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768521.525047:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768521.909034:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768522.549055:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768522.932976:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:3.0:1775768522.932978:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768522.932986:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1775768522.932989:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768522.932997:0:34107:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5354 00000100:00080000:3.0:1775768522.933007:0:34107:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768522.933013:0:34107:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:3.0:1775768522.933050:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740000 x1862023066672896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768522.933088:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768522.933092:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1775768522.933104:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768522.933114:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5354 00000100:00080000:3.0:1775768522.933118:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768522.933125:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:1.0:1775768522.933132:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066672896:192.168.202.105@tcp:250:kworker.0 00000008:00100000:3.0:1775768522.933134:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768522.933148:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188741c00 x1862023066673024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768522.933165:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768522.933168:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1775768522.933170:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768522.933174:0:34107:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5354 00000100:00080000:3.0:1775768522.933178:0:34107:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768522.933182:0:34107:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768522.933187:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1775768522.933198:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740380 x1862023066673152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1775768522.933207:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768522.933210:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1775768522.933211:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768522.933214:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5354 00000100:00080000:3.0:1775768522.933217:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768522.933220:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768522.933224:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:3.0:1775768522.933233:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188740700 x1862023066673280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768522.933241:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673024:192.168.202.105@tcp:38:kworker.0 00000100:00080000:3.0:1775768522.933241:0:34107:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1775768522.933245:0:34107:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1775768522.933248:0:34107:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1775768522.933252:0:34107:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5354 00000100:00080000:3.0:1775768522.933255:0:34107:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1775768522.933257:0:34107:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:3.0:1775768522.933260:0:34107:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:3.0:1775768522.933267:0:34107:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d8188742a00 x1862023066673408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768522.933279:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768522.933303:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768522.933324:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673408:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768523.573123:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768523.957006:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768524.597128:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768524.981010:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768525.622001:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768526.005038:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000400:00000100:2.0:1775768526.005199:0:10935:0:(lib-socket.c:454:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.105:988 00000400:00000100:2.0:1775768526.005238:0:10935:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.202.105@tcp at host 192.168.202.105 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1775768526.005254:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768526.005301:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768526.005308:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768526.005315:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000800:00000100:2.0:1775768526.005321:0:10935:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.105@tcp 00000100:00100000:1.0:1775768526.005398:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768522/real 1775768526] req@ffff9d8188740700 x1862023066673280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768526.005421:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740700 x1862023066673280/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768526.005434:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768526.005439:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768526.005444:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673280:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768526.005452:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768522/real 1775768526] req@ffff9d8188742a00 x1862023066673408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768526.005463:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188742a00 x1862023066673408/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768526.005475:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768526.005478:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768526.005481:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188742a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673408:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768526.005486:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768522/real 1775768526] req@ffff9d8188741c00 x1862023066673024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768526.005497:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188741c00 x1862023066673024/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768526.005504:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768526.005508:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768526.005511:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188741c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673024:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768526.005865:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768522/real 1775768526] req@ffff9d8188740380 x1862023066673152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768526.005924:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740380 x1862023066673152/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768526.005933:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768526.005936:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768526.005939:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673152:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768526.005946:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768522/real 1775768526] req@ffff9d8188740000 x1862023066672896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768526.005955:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d8188740000 x1862023066672896/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768526.005964:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768526.005966:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768526.005969:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d8188740000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066672896:192.168.202.105@tcp:250:kworker.0 00010000:00010000:3.0:1775768526.645035:0:34196:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768527.029074:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00010000:00010000:3.0:1775768527.669044:0:34195:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1775768528.053002:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 70783d78-a21c-41d0-8961-6627f7273664->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768528.053011:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1775768528.053015:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768528.053025:0:1374:0:(import.c:509:import_select_connection()) MGC192.168.202.105@tcp: connect to NID 192.168.202.105@tcp last attempt 5359 00000100:00100000:1.0:1775768528.053029:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds 00000100:00080000:2.0:1775768528.053036:0:1374:0:(import.c:588:import_select_connection()) MGC192.168.202.105@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768528.053055:0:1374:0:(import.c:623:import_select_connection()) MGC192.168.202.105@tcp: import ffff9d81892c0000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000100:00100000:2.0:1775768528.053087:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1180 x1862023066673536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768528.053113:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768528.053116:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1775768528.053117:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768528.053121:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5359 00000100:00080000:2.0:1775768528.053124:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768528.053128:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0000-mdc-ffff9d81892c2000: import ffff9d8190e60800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768528.053137:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1775768528.053147:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066673536:192.168.202.105@tcp:250:kworker.0 00000100:00100000:2.0:1775768528.053152:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1880 x1862023066673664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1775768528.053171:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768528.053174:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1775768528.053176:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768528.053180:0:1374:0:(import.c:509:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5359 00000100:00080000:2.0:1775768528.053185:0:1374:0:(import.c:588:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768528.053189:0:1374:0:(import.c:623:import_select_connection()) lustre-MDT0001-mdc-ffff9d81892c2000: import ffff9d8190e66000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768528.053193:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34681676 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1775768528.053205:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3800 x1862023066673792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768528.053208:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00080000:2.0:1775768528.053218:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768528.053221:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1775768528.053223:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1775768528.053227:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5359 00000100:00080000:2.0:1775768528.053231:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1775768528.053235:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0000-osc-ffff9d81892c2000: import ffff9d8190e64800 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768528.053240:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 8552448, lost: 0. 00000100:00100000:1.0:1775768528.053244:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768528/real 1775768528] req@ffff9d81984a1180 x1862023066673536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1775768528.053252:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a3100 x1862023066673920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1775768528.053260:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1180 x1862023066673536/t0(0) o250->MGC192.168.202.105@tcp@192.168.202.105@tcp:26/25 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1775768528.053262:0:1374:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1775768528.053269:0:1374:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1775768528.053272:0:1374:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1775768528.053274:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d81892c0000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1775768528.053277:0:1374:0:(import.c:509:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: connect to NID 192.168.202.105@tcp last attempt 5359 00000100:00080000:1.0:1775768528.053280:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.105@tcp failed (-110) 00000100:00080000:2.0:1775768528.053283:0:1374:0:(import.c:588:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1775768528.053284:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:70783d78-a21c-41d0-8961-6627f7273664:10943:1862023066673536:192.168.202.105@tcp:250:kworker.0 00000100:00080000:2.0:1775768528.053297:0:1374:0:(import.c:623:import_select_connection()) lustre-OST0001-osc-ffff9d81892c2000: import ffff9d8190e62000 using connection 192.168.202.105@tcp/192.168.202.105@tcp 00000008:00100000:2.0:1775768528.053302:0:1374:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640038 ocd_version: 34681676 ocd_grant: 5627904, lost: 0. 00000100:00100000:1.0:1775768528.053304:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:2.0:1775768528.053316:0:1374:0:(import.c:825:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9d81984a1500 x1862023066674048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1775768528.053323:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768528.053335:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768528/real 1775768528] req@ffff9d81984a1880 x1862023066673664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768528.053344:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1880 x1862023066673664/t0(0) o38->lustre-MDT0000-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768528.053349:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e60800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768528.053352:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768528.053354:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673664:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768528.053358:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673792:192.168.202.105@tcp:38:kworker.0 00000800:00000100:1.0:1775768528.053367:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768528.053372:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768528/real 1775768528] req@ffff9d81984a3800 x1862023066673792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768528.053379:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3800 x1862023066673792/t0(0) o38->lustre-MDT0001-mdc-ffff9d81892c2000@192.168.202.105@tcp:12/10 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768528.053383:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e66000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768528.053384:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768528.053386:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673792:192.168.202.105@tcp:38:kworker.0 00000100:00100000:1.0:1775768528.053389:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673920:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768528.053398:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768528.053403:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768528/real 1775768528] req@ffff9d81984a3100 x1862023066673920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768528.053408:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a3100 x1862023066673920/t0(0) o8->lustre-OST0000-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768528.053414:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e64800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768528.053415:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768528.053416:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066673920:192.168.202.105@tcp:8:kworker.0 00000100:00100000:1.0:1775768528.053424:0:10943:0:(client.c:1904:ptlrpc_send_new_req()) Sending RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066674048:192.168.202.105@tcp:8:kworker.0 00000800:00000100:1.0:1775768528.053432:0:10943:0:(socklnd_cb.c:976:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.105@tcp 00000100:00100000:1.0:1775768528.053437:0:10943:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1775768528/real 1775768528] req@ffff9d81984a1500 x1862023066674048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1775768528.053443:0:10943:0:(client.c:2510:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9d81984a1500 x1862023066674048/t0(0) o8->lustre-OST0001-osc-ffff9d81892c2000@192.168.202.105@tcp:28/4 lens 520/544 e 0 to 1 dl 1775768553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1775768528.053448:0:10943:0:(import.c:65:import_set_state_nolock()) ffff9d8190e62000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1775768528.053449:0:10943:0:(import.c:1506:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.105@tcp failed (-110) 00000100:00100000:1.0:1775768528.053451:0:10943:0:(client.c:2390:ptlrpc_check_set()) Completed RPC req@ffff9d81984a1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4eb5ffe8-b2bb-4904-b4d4-3375f25c82a1:10943:1862023066674048:192.168.202.105@tcp:8:kworker.0 00010000:00010000:3.0:1775768528.693001:0:34197:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1775768529.077086:0:34105:0:(client.c:2691:ptlrpc_set_wait()) set ffff9d81a04d4000 going to sleep for 0 seconds Debug log: 76580 lines, 76580 kept, 0 dropped, 0 bad.